viernes, 9 de diciembre de 2011

GIT Rev number on Python Package with SetupTool

Is very common to see on python projects a VERSION.py file or a VERSION string on setup.py. Every time the project has a new version, the you have to modify and push it. Personally, I don't like hardcoded version on the code, that responsibility should be share by the repository and the packager script. So after google it and not find something useful, I coded something, not so elegant, but works for me. Well first, setup your git repo and add tag in it. The way to get a cool revision number is:
git describe --abbrev=4 HEAD 
you will get something like :
0.0.1-29-gad35
Thats means, from the tag 0.0.1 there are 29 commits and the last one is gad35. In setup.py there is a field "version" so you can put a DEFAULT version mean while like '0.0.1' OR we can create a function that actually call the git command:
def get_version():
    git_call = subprocess.Popen(['git','describe','--abbrev=4','HEAD'], stdout=subprocess.PIPE)
    return git_call.stdout.read().strip()
and call it from the setup
setup( name='test'
    ,version=get_version()
    ,description=("Test blabla babla ........
That works, if the setup.py is called from the GIT repo. But what happen if it is called from the production environment, well will fail, because there is no GIT repo. So the change is kind of tricky.
def get_version(argv):
    if "install" in argv or "egg_info" in argv:
        #means setup.py is called from deployment process
        return "0.0.0"
    else:
        try:
            git_call = subprocess.Popen(['git','describe','--abbrev=4','HEAD'],
                                        stdout=subprocess.PIPE)
            return git_call.stdout.read().strip()
        except OSError:
            return "0.0.0"

The arg is added to the function, so the setup must provide them:
setup( name='test'
    ,version=get_version(sys.arg)
    ,description=("Test blabla babla ........
Now we need to create a packager script and get the revision number (from GIT or from the package created by setuptools). Then replace that REVISION number in the setup.py generated in the package. Repackage again and voila.
FILENAME=`python setup.py build sdist 2>/dev/null| grep Writing | cut -d' ' -f2 | cut -d/ -f1`

VERSION=${FILENAME#*lbsc-};
VERSION=${VERSION%/*}

pushd dist; tar xzf $FILENAME.tar.gz; rm $FILENAME.tar.gz;
pushd $FILENAME;
sed -i "s/return .0.0.0./return '$VERSION'/g" setup.py;
popd; tar czf $FILENAME.tar.gz $FILENAME;rm $FILENAME -rf;popd
Also we can create a get_version function on __init__.py and replace the VERSION on the code for the current one:
sed -i "s/^VERSION =.*$/VERSION = '$VERSION'/g" lbsc/__init__.py;
Now after install you can do:
import test; test.get_version()
Hope this was helpful.

miércoles, 13 de julio de 2011

Remove repeated elements from a list in Python

Say we have the python list:

>>> x = [5,6,5]

to remove the duplications, use the following code snippet:


>>> x_nodups = dict(map(lambda i: (i,1),x)).keys()


Taken from the desk of stinkpot

viernes, 18 de febrero de 2011

Procesando archivos en pararelo en bash con "xargs"

function process_file {
file=$1
do_something_with $file
}
export -f process_file


find PATH -type f | xargs --max-args=1 --max-procs=NUM -I{} bash -c process_file \{\}
or a shorter version
find PATH -type f | xargs -n1 -PNUM -I{} bash -c process_file \{\}



Where NUM is the number of parallel jobs and PATH where to start looking for the files.


source

jueves, 10 de febrero de 2011

creando una postgres db en ramdisk

#ramdisk de 5GB
mount -o size=5G -t tmpfs tmpfs /mnt/ramdb
chown -R postgres.postgres /mnt/ramdb

#en postgres ahora
CREATE TABLESPACE ramdb LOCATION '/mnt/ramdb';
CREATE DATABASE db WITH TABLESPACE ramdb;

viernes, 28 de enero de 2011

Plantilla de base de datos con PostGIS incluida

http://geospatial.nomad-labs.com/2006/12/24/postgis-template-database/


# Set postgis-1.5 path.
$ POSTGIS_SQL_PATH=`pg_config --sharedir`/contrib/postgis-1.5

# Creating the template spatial database
$ createdb -E UTF8 -T template0 template_postgis
# and add PLPGSQL language support.
$ createlang -d template_postgis plpgsql

# Loading the PostGIS SQL routines.
$ psql -d template_postgis -f $POSTGIS_SQL_PATH/postgis.sql
$ psql -d template_postgis -f $POSTGIS_SQL_PATH/spatial_ref_sys.sql

# Enabling users to alter spatial tables.
$ psql -d template_postgis -c "GRANT ALL ON geometry_columns TO PUBLIC;"
$ psql -d template_postgis -c "GRANT ALL ON geography_columns TO PUBLIC;"
$ psql -d template_postgis -c "GRANT ALL ON spatial_ref_sys TO PUBLIC;"

# Garbage-collect and freeze.
$ psql -d template_postgis -c "VACUUM FULL;"
$ psql -d template_postgis -c "VACUUM FREEZE;"

# Allows non-superusers the ability to create from this template.
$ psql -d postgres -c "UPDATE pg_database SET datistemplate='true' WHERE datname='template_postgis';"
$ psql -d postgres -c "UPDATE pg_database SET datallowconn='false' WHERE datname='template_postgis';"

viernes, 17 de diciembre de 2010

Cambiando formato de timestamp con awk

Una linea con epoch time, latitud y longitud.

El epoch time se esta reemplazando por la fecha en formato "%Y%m%dT%H%M%S"

echo "1288552753,38.995177,-76.881886" | \
TZ=UTC awk -F, '{ print strftime("%Y%m%dT%H%M%S",$1)","$2","$3 }'