Linux ReadyBoost alternative aka "how to cache a slow big drive on another quick drive"

I have looked forward a way to cache a big slow drive on another quick drive (in a similar way as Windows7 ReadyBoost already do with flash usb drives).
I have found out this interesting project, flashcache (GitHub) which is a facebook released code;
Flashcache works at the block device level and take advantege of quick SSD drives.
Microsoft ReadBoost caches little files by taking advantage of slower flash device (access time of slower flash device is still lower than hard disk one).
So, ReadyBosst and flashcache are similar but different.
I'll surely try flashcache in the future but it's a shame it doesn't come bundled with vanilla kernels.

More infos here: Facebook FlashCache Post

Self compile Tesseract 3.0.1 on Debian Squeeze, libleptonica problem

In my case, ./configure fails with an error :


checking for asin... yes
checking for leptonica... yes
checking for pixCreate in -llept... no
configure: error: leptonica library missing

Libleptonica package was correctly installed but something is broken; there are tons of workarounds around about setting paths or other magic options, they are useless.
In order to solve the issue you have to uninstall (by apt-get) the debian libleptonica package and self compile libleptonica by source; After libleptonica self installation (make install), tesseract compilation 'll work as expected.
Get leptonica source.
Get Tesseract compialtion instructions.

Openvpn server with Qnap TS-409 Pro, you can

 
Qnap ts-409 Pro is an old but powerful 4-bay NAS box (linux ARM) and comes with a nice proprietary webinterface and lot of functionalities.

The Mission

Add an OpenVPN access to my home network.

I like to have full access to all my home machines (bonus target: grant the home machines access to the VPN subnet as well).

The Plan

Installing openvpn on the main router/firewall is usually a good way to go but my home network has, as usual, a low-end TP-Link ADSL router and there is no way to install openvpn into that device.

The plan is to install the openvpn server instance into my only 24h/24h server, the Qnap TS-409 Pro. 

Warning

Installing a new software, changing configuration files and writing into flash are, for inexperienced users, always dangerous operations.

This guide may apply to other similar Qnap NAS models as well but you have to check for correctenss and applicability of this procedure by your own.

I'm not responsible of any damage, if you brick your box it's your fault, not mine.

The Solution

The Qnap community is very active and the box comes with a lot of QPKG packages, e.g. installing OpenWRT/ipkg support is straightforward.

This is the step by step guide I have used; here is the summary:

1. QPKG Optware installion
2. ipkg update
3. ipkg install of openvpn package
4. manual installation of tun.ko module
5. create CA and certificates (EasyRSA, you can do this your pc)
6. create server configuration files
7. create client configuration files
8. set port forward on router
9. configure new static route on the router
10. enable network forwarding on the TS-409
11. test configuration
12. change autostart.sh in order to persist changes thru reboot

It is mandatory to apply these  OpenVPN_Extras instructions (9,10) in order VPN to communicate properly with LAN; The tun device works only by routing properly the traffic.

In next paragraphs I'm not going to rewrite the linked wiki but I'll focus only on the tricky and interesting parts.

Details

Assuptions and network topolgy

Details about my network topology and choices about openvpn service ports.

Local network IP range: 192.168.55.0/24 (Subnet 255.255.255.0)
Qnap NAS server, home LAN IP: 192.168.55.77

OpenVPN service listens to 192.168.55.77:22111
ADSL Router forwards (Virtual Server) incoming UDP WAN traffic to port 7788 to the home host 192.168.55.77:22111 (yes, different port)

Clients will connect to Openvpn thru an external port which looks different from 22111, my choice

Server configuration

Mount the flash device in order to edit autorun.sh, with command:
mount -t ext2 /dev/mtdblock5 /tmp/createThisDirectoy/

mtdblock5 is the right device name for Qnap TS-409 Pro, check your box guide or wiki in order to find out your own.

... pay attention, after hacking the flash please umount in order to save changes properly:
umount /tmp/createThisDirectoy/

My autorun.sh looks like this, this chages are needed in order to restart openvpn automatically after a system shutdown

mkdir /dev/net;
mknod /dev/net/tun c 10 200;
(sleep 10;insmod /opt/etc/openvpn/modules/tun.ko)&

echo "1" > /proc/sys/net/ipv4/ip_forward

(sleep 15; /opt/sbin/openvpn /opt/etc/openvpn/easy.conf)&

The green row enables ip forwarding inside TS-409, this will route the VPN traffic from the tun interface towards the home lan interfaces, and the box will forward the packets to the destination machines.

my current easy.conf looks like this:

# OpenVPN server configuration QNAP NAS
# basic settings
# you have to choose your port (the listen port of openvpn service)
port 22111
proto udp
dev tun
#
# detect mtu if the connection is slow.
; mtu-test
#
# define mtu, if necessary
; tun-mtu xyz
#
# define the ip-addresses of the underlying tunnel
server 10.8.5.0 255.255.255.0
#
# Route
push "route 192.168.55.0 255.255.255.0"   #  <--- Enter the ip-address of your home network here!
#
# certificates & keys
dh /opt/etc/openvpn/keys/dh2048.pem
ca /opt/etc/openvpn/keys/ca.crt
cert /opt/etc/openvpn/keys/server.crt
key /opt/etc/openvpn/keys/server.key
#
# data compression
comp-lzo
#
# allow, that several clients with the same common name log on
; duplicate-cn
#
# different clients can "see" each other through the tunnel.
; client-to-client
#
# Keepalive
keepalive 15 120
#
# verbosity of status messages in the console. Activate for debugging (1-9) possible)
; verb 5
#
# Log files
status /share/HDA_DATA/.qpkg/Optware/etc/openvpn/log/status.log
log-append /share/HDA_DATA/.qpkg/Optware/etc/openvpn/log/openvpn.log
#
# Run as daemon (activate, after everything is set up properly)
; daemon
#
# Management Interface. Access with "telnet localhost 7505"
management localhost 7505

Notes:
The TS-409 Pro Optware folder is located (by QPKG installer) in a path which differs from the wiki guide, in my case my path is: /share/HDA_DATA/.qpkg/Optware/
Let's check optware path and files location on your box and change the script accordingly.

Client configuration

The clients, with the proper private keys and certificates, connect to the OpenVPN server using this configuration file:

# connect to QNAP OpenVPN Server
# 
port 7788
proto udp
dev tun
tls-client
remote youraccountname.dynds.net 7788  #  <--- enter your dyndns-account here!
pull
# set mtu, if necessary
; tun-mtu xyz
#
resolv-retry infinite
nobind
persist-key
persist-tun
# certificates and keys
# Note the double \\ in the path for a windows config
ca ca.crt
cert your.crt
key your.key

ns-cert-type server
verb 4
comp-lzo


There are redundant settings and useless options too but, you know, this configuration file works :-)


Router configuration

The router has to do two operations:

• forwards (Virtual Server) incoming UDP traffic to port 7788 to 192.168.55.77:22111
• Routes all the packets with destination 10.8.5.0/24 to the gateway 192.168.55.77

The first one is a basic router configuration, usually named "virtual server", the second one is a bit difficult and you net to set a static route into your router.
The rule is this:
  

Destination	Subnet Mask	Gateway	Interface	x
10.8.5.0	255.255.255.0	192.168.55.77	br0

 br0 is the internal LAN bridge.

Why this route?

This static route is necessary for the LAN to send back packets to the VPN.. the LAN's machines are going to send all the VPN packets to the router (the destination is another subnet) and without this rule they are going to be lost o rejected.

Conclusions

I have only changed few details from the original wiki and I have succefully achived my objective.
Being able to connect to my home network thru VPN, being able to connect and communicate to all machines inside.

OO-GC Language for standalone executables (Part 2)

This is an update on the topic.
In the first part I have presented my findings about few languages and tools, hopefully I have inspired a talented friend of mine (I'll refer to him as The Amazing Vlad,or simply The Amazing)
The Amazing works with C#/mono and has tested the Mono's mkbundle for real (in order to create a standalone executable); mkbundle isn't working well as expected, so I have removed mono from the winners list.
Here I paste the translated friend's mail:


Hi,

I am looking torward creating a portable version of KeePass for Linux.

I have installed mono 2.1x on CentOS5 and I have tried creating a bundle for KeePass by using --deps --static.

The first option might bundle all the needed managed assemblies,  the sedcond one might statically link the native mono library. (lib_mono_something...)

It turns out that mkbundle doesn't link the other needed native libraries, e.g. libgdiplus  which is called by WinForms applications.

I have tried using the generated bundle on CentOS6, and I have faced these problems:

 1

It is impossibile to have a priori knowledge (using ldd command) about dynamic library dependencies because managed assemblies load them at runtime .e.g. libc. 

Ldd has stated libc as a dependency but something inside the bundle looked for libc.so which it doesn't exist. The error arises because the real name is libc.so.6.

But how does Mono know about the real name of libc library? Quite simple, mono has a configuration file with a number of mapping directives for windows dll and dynamic linux libraries.

I have fixed the issue by copying and renaming the config as NameExecutable.exe.config  and I have forced mkbundle to add this file to the bundle.

2

The program still has kept failing and ranting about lacking libgdiplus. As mentioned above, no native dynamic library is bundled with mono (by using --static). So, you are forced copying all the needed libraries with your application (plus their own dependencies).

I have copied libgdiplus.so.x and It still failed cause lacking MonoFuckedLongName.dll which doesn't exist in my system!!

I gave up

NOTE: A simple commandline Hello World application still works flawless, with --deps --static ; no tricks or workarounds are needed.

Conclusion

Mkbundle doesn't work well with complex applications and lot of libraries depencies, but it might work well for simple commandline applications. Perhaps WinForms might work by copying libgdiplus with your application but I have not tested a graphical hello world application yet.

 The Amazing

Machine Learning Lab - IngInf@UniTS: Automatic generation of regular expressions

My colleagues at the Machine Learning Laboratory (University in Trieste) have developed a demo of their great engine for the automatic generation of regular expressions based on genetic programming.
Let's try this web application and have fun! (if you are  Regex-addicted as I am)

Machine Learning Lab - IngInf@UniTS: Automatic generation of regular expressions: Today we launched the online prototype of our automatic regular expressions generator! Take a look at regex.inginf.units.it , all comments ...

Or click on the direct link http://regex.inginf.units.it/

OO-GC Language for standalone executables

Recently I have looked toward an object oriented-garbage collected programming language, able to build a standalone executable. The reason is simple, I'm in need to deploy applications on Linux servers by minimizing installation requirements.

Performance and portability are highly appreciated features

JAVA

Great language, high performance, big user base and lot of resources but there is no way to bundle a minimal VM and dependencies in a single executable, so ,you are in need to move the full framework around with your tiny console application.

There is also the GCJ which provides a java compiler but it looks unmaintained and there are no updates since 2009. Using GCJ looks more like an hack to me but further inspection is needed.
UPDATE: This project, Avian is a "lightweight Java VM and class library designed to provide a useful subset of Java's features, suitable for building self-contained applications"[Wikipedia]. This looks like the way to build a standalone executable from current Java code. Java + Avian looks like a second (au pair) winner.
I hope to try out this soon.

OCAML (Objective CAML)

It looks promising.

Tricky language which merges functional and imperative paradigms and has the garbage collection; Ocaml builds native binaries or creates an executable which bundles together a minimal Ocaml's VM and application bytecode; Ocaml single thread performance is quite good both with native and bytecode applications.

Base Ocaml and 3th-party developers provides a fair amount of libraries.

The Ocaml cons are that it doesn't support real multithreading (you have to use 3-party libraries, but it looks just an hack).

Another issue is about being a bit unusual (functional style programming is not the first choice) language with a little user base.

C# (MONO)

It looks more promising.

Mono works with well-known OO languages (I like C#), large user base, garbage collector, very large standard library and lots of 3th-party libraries and there are lot of resources around.

Good single thread performance as well as support for real multithreading.

About having a standalone executable, you can make it by using mkbundle.(I have to check it out)

It looks like the winner. updated in Part2

GO

Last but not least, this language is an interesting new entry and the reason behind this post; One year ago I have thought about this new language being useless, I was wrong.

At the moment I think I have understood two reasons behind the Google Go project:

• provide an (about) OO, garbage collected, language which compiles to a native executable
• provide a new approach to OO programming, Go doesn't provide inheritance or method overloading  but enforces interfaces as unique alternative.

Go developers state that inheritance is a dangerous feature; you know, inheritance is not about code reuse but is about abstraction, about sharing functionalities between conceptually similar objects and about taking advantage of polymorphism but inheritance is sometimes... misused.

Code reuse is still important and it is possible with Go by using composition and interfaces  (instead of inheritance). This is intriguing.

I think that the GO language deserves further inspection.

There are also big cons, it is a young language with a little library base and very tiny user base. But I have added "Go" as one of my Schemer Tasks for a Brighter Future!

It doesn't fit with the topic but I also suggest my readers to have a look at this interesting article programming language trends.

This post is a start point, I think I'll go deeper into Mono,GO and have a look at gcj but at the moment I'm highly oriented toward Mono. 

I'll let you updated on the topic.

Further details in Part 2

SafeNet Authentication Client , how to solve installation issues

Recently I have had problems with a SafeNet Authentication Client v.8.0.0 installation  on Windows7 SP1, the SafeNet installation was simply broken and I wasn't able to repair or uninstall ( failed with a generic error ) and  the installer don't permit to do a fresh install over an existing one.

 I tried CCleaner in order to remove the application entry but the CCleaner failed with an error : "Impossible to delete MSI.."

I was sticked.

The solution is

1. Use  http://en.wikipedia.org/wiki/Windows_Installer_CleanUp_Utility in order to delete the installation information from your system. Please, pay attention, let's select and delete ONLY the SafeNet Authentication Client entry!! No rollback exists for this operation. Alternate download link.
2. You are able to do do a fresh SafeNet install, but the installation still fails with an "error: 1402". The error contains details about registry keys which the installer is not able to read/write.. yes it is a permission issue. The solution is here: http://www.okino.com/conv/changing_windows_registry_permissions.htm , abstract: you have to edit execute regedit as adminsitrator, find out the root of the problematic keys and change the permissions as pointed in the linked article.
3. You have to iterate the installation procedure 2) as far as you have fixed permissions an all the keys.

Hope useful,
Good luck

Recupero dati con TestDisk

Storia di un recupero dati..

Problema

MI è stato chiesto di recente di recuperare dei dati da un disco rigido sata da 500G; per windnows il disco risultava vuoto e chiedeva, come prevedibile di formattarlo.




Prima analisi

Da linux ho cercato di ricavare qualche info con il comando "fdisk -l", scoprendo che il disco conteneva una partizione estesa al cui interno si trovava una FAT16 da 500G!!

Alla fonte

Inquisendo il proprietario si è scoperto che un amico dell'amico aveva messo questo disco su un pc di provenienza XYZ e che il PC lo aveva accettato solo configurando il controller RAID in modo da tenere il disco come unico componente di un volume jbod. JBOD!
Naturalmente il PC in questione è sparito in una voragine, nulla si sa di lui e non esiste più.

Il manufatto atlantideo svelato

Avevo quindi a che fare con un "hardware jbod" di un pool costituito da un solo disco, ma ero ottimista; il raid1 hardware spesso non crea problemi e il singolo disco inserito in un PC anche in configurazione non raid risulta generalmente accessibile ed usabile, essendo il JBOD una mera concatenazione di dischi speravo che la partizione NTFS fosse li, integra, liscia, intonsa, al massimo preceduta da immondizia lasciata dal controller.

Fase 0

Essendo un malfidente e un pessimista cronico ho prima effettuato da Linux una copia dell'MBR e quanto di vitale, quotando http://www.partimage.org/Partimage-manual_Backup-partition-table (riporto esempio, differisce dal mio caso dove il discpositivo è /dev/sdc):

dd if=/dev/hda of=backup-hda.mbr count=1 bs=512
It will produce a very small, but very important file: 512 bytes of data. Now, we will save entries of the extended partitions:
sfdisk -d /dev/hda > backup-hda.sf

questi files sono necessari per ripristinare la struttura delle partizioni in caso di errori

Tentativo 1, montare il JBOD su Linux

Il primo tentativo era verificare (più per curiosità) se il formato in qualche modo coincideva con il --linear del mdadm di Linux (che da quanto sapevo dovrebbe corrispondere al jbod).
Naturalmente Mdadm (mdadm -Q /dev/sdc) dice che non c'è nulla, ma ero pessimista pure io.

Tentativo 2

Vari tentativi con utility Windows(falliti)

Tentativo 3 (che invero sapevo che doveva essere il primo) l'utility TestDisk

Questa prodigiosa utility, sommo taumaturgo, ineguagliabile panacea, è sicuramente l'implementazione informatica di sciamanici riti di guarigione.
Dopo aver scaricato da http://www.cgsecurity.org/wiki/TestDisk_Download la versione 6.13 sulla mia fedele OpenSuse 11.2 e eseguito testdisk come root, mi è bastato selezionare il device (nel mio caso /dev/sdc) e lanciare la "Quick Analisys".

Figura di esempio, non legata al caso specifico

TestDisk ha subito identificato un unica partizione NTFS da 500G!!
Certo di lavorare in sicurezza, grazie al backup già fatto nella Fase0 dell'MBR, ho fatto scrivere la nuova tabella a TestDisk sul disco con il comando "Write" (Oh dolce Semplicità, a volte uno dimentica quanto sei bella!)
Finito bisogna dire a linux di caricare la nuova tabella delle partizioni, con il seguente comando (/dev/sdc nel mio caso è il nome del device che stavo recuperando):
partprobe /dev/sdc
A questo punto resta da verificare il successo!
Il resto è discesa, riavviare in windows ed effettuare un chkdsk (readonly, ovvero senza l'opzione /F) per verificare che tutto sia a posto.
Anche il comando smartctl utilizzato da linux conferma che il disco è vetusto ma non imploso.
Windows è in grado di montare la nuova partizione e di accedere ai files, perfettamente integri.
Finito in 10 minuti.

Conclusioni

TestDisk è una grande applicazione per il recupero delle partizioni e fa molto, molto di più di quanto vi ho raccontato. Tenetevela sempre a portata di mano(e mi pare sia anche inclusa in systemrescuecd).


Note per i posteri

Il recupero dati non si fa così , non ci si accontenta mai del semplice backup dell'MBR e poi lavorando direttamente sul disco originale ma si deve sempre fare una copia raw con ddrescue e creare un file immagine del disco.

Tutte le operazioni successive dovrebbero essere fatte su questa copia perché le operazioni di recupero sono operazioni di lettura/seek/scrittura molto intense ed è possibile che il disco decida di mollarvi durante l'operazione; ddrescue in questo caso è un tool grandioso e gestisce in modo intelligente i bad sector.

Nel mio caso ho deciso di lavorare a cuore aperto per più motivi:

1. non avevo un posto dove mettere 500G di immagine
2. sapevo dalle info Smart e dal racconto che il disco non era fisicamente danneggiato(la causa del malfunzionamento era l'estrazione dal PC atlantideo)
3. non avevo molto tempo a disposizione
4. c'erano foto ma il valore dei dati non era altissimo (alias non erano dati miei :-) )