Book Image

Geospatial Development By Example with Python

By : Pablo Carreira
5 (1)
Book Image

Geospatial Development By Example with Python

5 (1)
By: Pablo Carreira

Overview of this book

From Python programming good practices to the advanced use of analysis packages, this book teaches you how to write applications that will perform complex geoprocessing tasks that can be replicated and reused. Much more than simple scripts, you will write functions to import data, create Python classes that represent your features, and learn how to combine and filter them. With pluggable mechanisms, you will learn how to visualize data and the results of analysis in beautiful maps that can be batch-generated and embedded into documents or web pages. Finally, you will learn how to consume and process an enormous amount of data very efficiently by using advanced tools and modern computers’ parallel processing capabilities.
Table of Contents (17 chapters)
Geospatial Development By Example with Python
Credits
About the Author
About the Reviewers
www.PacktPub.com
Preface
Index

Importing massive amount of data


Now that our environment is ready, we can begin working with bigger datasets. Let's start by profiling the import process and then optimize it. We will start with our small geocaching dataset and after the code is optimized we will move to bigger sets.

  1. In your geodata_app.py file, edit the if __name__ == '__main__': block to call the profiler.

    if __name__ == '__main__':
        profile = cProfile.Profile()
        profile.enable()
        import_initial_data("../data/geocaching.gpx", 'geocaching')
        profile.disable()
        profile.print_stats(sort='cumulative')
  2. Run the code and see the results. Don't worry about duplicated entries in the database now, we will clean it later. (I removed some information from the following output for space reasons.)

    Importing geocaching...
    112 features.
    Done!
     1649407 function calls (1635888 primitive calls) in 5.858 seconds
    
    cumtime  percall filename:lineno(function)
      5.863    5.863 geodata_app.py:24(import_initial_data)
      5.862    5.862...