Posts

Showing posts from April, 2018

How to convert kml to shape format throught ogr2ogr

You can create your kml file online with scribblemaps and download it to your computer.
If you create lines and markers in the same file, you will get this error:

ogr2ogr -f "ESRI Shapefile" file.shp file.kml -skipfailures
Warning 6: Normalized/laundered field name: 'description' to 'descriptio'
Warning 6: Field timestamp create as date field, though DateTime requested.
Warning 6: Field begin create as date field, though DateTime requested.
Warning 6: Field end create as date field, though DateTime requested.
Warning 6: Normalized/laundered field name: 'altitudeMode' to 'altitudeMo'
ERROR 1: Attempt to write non-point (LINESTRING) geometry to point shapefile.
ERROR 1: Attempt to write non-point (LINESTRING) geometry to point shapefile.
ERROR 1: Attempt to write non-point (LINESTRING) geometry to point shapefile.

To avoid it, you must create separated files for points, linestring or polygon as:

ogr2ogr point.shp point.kml -skipfailures -nlt point
ogr2ogr …

How to find the shortest path from an Internet Zoo topology file throught Postgis/pgrouting

Image
First, you need to download a topology from Internet zoo.
I will use Napnet as example:

wget http://www.topology-zoo.org/files/Napnet.graphml

Now, we need to convert graphgml to sql (to load from postgres). To do it, you can use simplekml library for python or pykml.
It's important use geographic coordinates without altitude value and only write edges in the kml file.

After, you can follow this post to convert from kml to sql.

To begin, first we need to create a database and create a password (usually is postgres) for postgres user:

sudo -u postgres createdb testing
sudo -u postgres psql  

postgres=# \password postgres
Enter new password: 
Enter it again: 

postgres=# \q

Now, load the sql query:

psql -U postgres -d testing -q -f "Napnet.sql"
psql: FATAL:  Peer authentication failed for user "postgres"

To avoid this issue, you can run it throught this way:

psql 'host=localhost port=5432 dbname=testing user=postgres' -q -f "Napnet.sql"
psql:Napnet.sql:17: …

How to install Qgis 3.0.x in Ubuntu 18.04

Image
If you try to install from repositories, you will install version 2.18
To install the lastest version, you can download it from git, but first you need to install dependencies.
If you have Ubuntu 18.04 you need the following packages:

sudo apt-get install bison ca-certificates ccache cmake cmake-curses-gui dh-python doxygen expect flex gdal-bin git graphviz grass-dev libexpat1-dev libfcgi-dev libgdal-dev libgeos-dev libgsl-dev libosgearth-dev libpq-dev libproj-dev libqca-qt5-2-dev libqca-qt5-2-plugins libqt5opengl5-dev libqt5scintilla2-dev libqt5sql5-sqlite libqt5svg5-dev libqt5webkit5-dev libqt5xmlpatterns5-dev libqwt-qt5-dev libspatialindex-dev libspatialite-dev libsqlite3-dev libsqlite3-mod-spatialite libzip-dev lighttpd locales ninja-build pkg-config poppler-utils pyqt5-dev pyqt5-dev-tools pyqt5.qsci-dev python3-all-dev python3-dateutil python3-dev python3-future python3-gdal python3-httplib2 python3-jinja2 python3-markupsafe python3-mock python3-nose2 python3-owslib python3-plotly…

How to run Cplex throught Julia

Image
In my last post I talk about how to run Gurobi throught Julia.
To do a benchmarking, now I going to explain how to run Cplex.
First, you need to register here to download cplex



chmod +x cplex_studio128.linux-x86-64.bin 
 ./cplex_studio128.linux-x86-64.bin 

By default, cplex use /opt/ibm/ILOG/CPLEX_Studio128 to install it

After, you need to add the binaries to your PATH or create symbolic links to the executables:

sudo ln -s /opt/ibm/ILOG/CPLEX_Studio128/cplex/bin/x86-64_linux/cplex /usr/bin/cplex
sudo ln -s /opt/ibm/ILOG/CPLEX_Studio128/cplex/bin/x86-64_linux/cplexamp /usr/bin/cplexamp

cd /opt/ibm/ILOG/CPLEX_Studio128/cplex/python/3.6/x86-64_linux
sudo python3 setup.py install

Now, add Cplex package to Julia:

julia> Pkg.add("CPLEX")
INFO: Cloning cache of CPLEX from https://github.com/JuliaOpt/CPLEX.jl.git
INFO: Installing CPLEX v0.3.2
INFO: Building CPLEX
================================[ ERROR: CPLEX ]================================

LoadError: Unable to locate CPLEX installation. …

How to run Gurobi throught Julia

To install Julia, we can download it from github

git clone https://github.com/JuliaLang/julia.git
cd julia
sudo apt-get install build-essential libatomic1 python gfortran perl wget m4 cmake pkg-config
make -j8
make install
cd julia-*
cd bin
pwd

Now, export the working directory to your .bashrc file

export PATH="$(pwd)/julia:$PATH"

Run julia and add JuMP (Julia for Mathematical Programming):

Pkg.add("JuMP")
WARNING: Base.Pkg is deprecated, run `using Pkg` instead

Newer versions must use Pkg first as following:

using Pkg
Pkg.add("JuMP")

Other option is install it from the tarball file:

wget https://github.com/JuliaLang/julia/releases/download/v0.6.2/julia-0.6.2-full.tar.gz
tar xzvf julia-0.6.2-full.tar.gz
cd julia-0.6.2
make -j8
make install
cd julia-*
cd bin
pwd

Export to the bashrc file again and install the package:

Pkg.add("JuMP")
Pkg.add("Gurobi")
Pkg.update()

Maybe you can have issues to load gurobi if the environment is not configured:

using Gurobi
INFO: P…

How to install gurobi in Ubuntu 18.04 outside an academic domain

Image
To install gurobi, you can do it throught anaconda or tarball, secondly we need configure PATHS and  register the licence.

Anaconda option to install:

wget https://repo.continuum.io/archive/Anaconda3-5.1.0-Linux-x86_64.sh
chmod +x Anaconda3-5.1.0-Linux-x86_64.sh 
./Anaconda3-5.1.0-Linux-x86_64.sh 
conda update -n base conda
conda update -n base conda
conda config --add channels http://conda.anaconda.org/gurobi
conda install gurobi

If you want to uninstall anaconda, you can do it the following:

conda install anaconda-clean
anaconda-clean
rm -rf ~/anaconda3

Tarball option to install:

First, you need to create an account in Gurobi here and after proceed to download it:

wget http://packages.gurobi.com/7.5/gurobi7.5.2_linux64.tar.gz
tar xzvf gurobi7.5.2_linux64.tar.gz
cd gurobi752/linux64
sudo python setup.py install

Configure gurobi:

We need to add the PATH to our environment. By default, in linux, we use .bashrc
Edit ~/.bashrc and add the following lines:

export GUROBI_HOME="/path_where_you_unzip_…

How to upgrade pgrouting to lastest version (2.6)

If you installed pgrouting from synaptic, probably you have pgrouting 2.4.2.
To upgrade to the last version, download it from github:

wget https://github.com/pgRouting/pgrouting/releases/download/v2.6.0/pgrouting-2.6.0.tar.gz
tar xzvf pgrouting-2.6.0.tar.gz
cd pgrouting-2.6.0
mkdir build
cd build
cmake ..

-- POSTGRESQL_PG_CONFIG is /usr/bin/pg_config
-- POSTGRESQL_EXECUTABLE is /usr/lib/postgresql/9.6/bin/postgres
-- POSTGRESQL_VERSION_STRING in FindPostgreSQL.cmake is PostgreSQL 9.6.8
-- PostgreSQL not found.
CMake Error at CMakeLists.txt:294 (message):
   Please check your PostgreSQL installation.

To solve it, you need to install postgres developer

sudo apt-get install postgresql-server-dev-9.6
cmake ..

Could not find the following Boost libraries:

          boost_thread
          boost_system

-- CGAL not found.
CMake Error at CMakeLists.txt:351 (message):
   Please check your CGAL installation, or set correct path to CGAL_INCLUDE_DIR and CGAL_LIBRARIES.

To solve it, you need to install cgal libraries

Load kml geojson osm shp files in Postgis

In my last post I talk how to convert from kml to GeoJson
Now, I want to load this file in a Spatial and Geographic Objects for PostgreSQL (postgis)
I read in many sites that you can convert from osm 2 postgres, then I will try to convert to osm file with geojsontoosm.

# sudo  npm install -g geojsontoosm

npm ERR! enoent ENOENT: no such file or directory, open '/tmp/npm-15537-43d2f472/unpack-edff7974/package.json'
npm ERR! enoent ENOENT: no such file or directory, open '/tmp/npm-15537-43d2f472/unpack-edff7974/package.json'
npm ERR! enoent This is most likely not a problem with npm itself
npm ERR! enoent and is related to npm not being able to find a file.
npm ERR! enoent 

To install, I used the git files:

# git clone https://github.com/tyrasd/geojsontoosm.git
# cd geojsontoosm

# ./geojsontoosm 
module.js:471
    throw err;
    ^

Error: Cannot find module 'jxon'
Error: Cannot find module 'optimist'
Error: Cannot find module 'geojson-stream'
Error: Cannot find mod…

How to fix npm

I tried to install togeojson, but I got the following error:

npm install -g @mapbox/togeojson 

npm ERR! Linux 4.4.0-116-generic
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "install" "-g" "@mapbox/togeojson"
npm ERR! node v8.0.0-pre
npm ERR! npm  v3.5.3
npm ERR! code MODULE_NOT_FOUND

npm ERR! Cannot find module 'internal/fs'
npm ERR! 
npm ERR! If you need help, you may report this error at:
npm ERR!     <https://github.com/npm/npm/issues>

npm ERR! Please include the following file with any support request:
npm ERR!     /home/nboettcher/Downloads/togeojson/npm-debug.log

Other common error is:

npm ERR! Cannot find module 'ansi'

To solve them, you must reinstall nodejs and npm with the following sentences:

sudo apt-get remove nodejs npm
curl -sL https://deb.nodesource.com/setup_6.x | sudo -E bash -
sudo apt-get install -y nodejs
sudo apt-get install npm
npm install

Now I have npm 3.5.2, but now it works.
Enjoy it!

How to convert from kml to GeoJson with python3

To do it, you can import ogr from osgeo

from osgeo import ogr

def kml2geojson(kml_file):
    drv = ogr.GetDriverByName('KML')
    kml_ds = drv.Open(kml_file)
    for kml_lyr in kml_ds:
        for feat in kml_lyr:
            print feat.ExportToJson()

kml2geojson('/path/to/your.kml')

But, if you try to run it, maybe you will see the following error:

ImportError: No module named 'osgeo'

This means that you need gdal libraries. If you try to install it with pip3 you will see the following:

pip3 install gdal
...
cc1plus: warning: command line option ‘-Wstrict-prototypes’ is valid for C/ObjC but not for C++
    extensions/gdal_wrap.cpp:3168:22: fatal error: cpl_port.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

We search which pakage includes cpl_port.h:

apt-file search cpl_port.h   
libgdal-dev: /usr/include/gdal/cpl_port.h

After install python3-gdal and libgdal-dev you will see that you get the…

How to run pykml in python3

First, you need to install pykml for python3:

sudo pip3 install pykml

Now, if you try to read a kml file you will get the following message:

Python 3.6.3 (default, Oct  3 2017, 21:45:48) 
[GCC 7.2.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from pykml import parser
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.6/dist-packages/pykml/parser.py", line 8, in <module>
    import urllib2
ModuleNotFoundError: No module named 'urllib2'

To solve it, you need to fix urllib2 to work well in python3

If you try to install it with pip3 you will get:

Collecting urllib2
  Could not find a version that satisfies the requirement urllib2 (from versions: )
No matching distribution found for urllib2

To solve it, you must edit /usr/local/lib/python3.6/dist-packages/pykml/parser.py and replace:

import urllib2

to

try:
    # For Pyt…