Sunday, October 23, 2011

Finding the Maximum using Relational Algebra

We know that if you want to get the maximum value of a column in SQL, you can simply use the MAX function as shown below:
SELECT MAX(value) FROM T
You can also do it without the MAX function as follows:
SELECT T.* FROM T
MINUS
SELECT T.* FROM T, T as T2 WHERE T.value<T2.value
or:
SELECT T.* FROM T
LEFT JOIN T as T2 ON T.value<T2.value
WHERE T2.value IS NULL
Relational Algebra:
Using Relational Algebra (RA) syntax, this would be:
\project_{value}(T)
\diff
\project_{value} (
    \select_{value < value2}(
      \project_{value}(T)
      \cross
      \rename_{value2}(\project_{value}(T))
    )
)
where:
  • \cross is the relational cross-product operator
  • \diff is the relational diff operator
  • \project_{attr_list} is the relational projection operator
  • \rename_{new_attr_name_list} is the relational renaming operator
  • \select_{cond} is the relational selection operator

Sunday, October 16, 2011

Validating XML with xmllint

The following commands show you how to validate an XML file against a DTD or XSD using xmllint.

To validate an XML file against:

  • a DTD stored in the same file:
  • xmllint --valid --noout fileWithDTD.xml
    
  • a DTD stored in a separate file:
  • xmllint --dtdvalid DTD.dtd --noout fileWithoutDTD.xml
    
  • an XSD stored in a separate file:
  • xmllint --schema schema.xsd --noout file.xml
    
The --noout option suppresses the output of the xml file.

Example
To validate:

<countries>
  <country name="Afghanistan" population="22664136" area="647500">
    <language percentage="11">Turkic</language>
    <language percentage="35">Pashtu</language>
    <language percentage="50">Afghan Persian</language>
  </country>
  <country name="Albania" population="3249136" area="28750"/>
  <country name="Algeria" population="29183032" area="2381740">
    <city>
      <name>Algiers</name>
      <population>1507241</population>
    </city>
  </country>
</countries>
against:
<!ELEMENT countries (country*)>
<!ELEMENT country (language|city)*>
<!ATTLIST country name CDATA #REQUIRED>
<!ATTLIST country population CDATA #REQUIRED>
<!ATTLIST country area CDATA #REQUIRED>
<!ELEMENT language (#PCDATA)>
<!ATTLIST language percentage CDATA #REQUIRED>
<!ELEMENT city (name, population)>
<!ELEMENT name (#PCDATA)>
<!ELEMENT population (#PCDATA)>
use the command:
xmllint --dtdvalid countries.dtd --noout countries.xml

Saturday, October 08, 2011

Splitting a large file into smaller pieces

If you have a large file and want to break it into smaller pieces, you can use the Unix split command. You can tell it what the prefix of each split file should be and it will then append an alphabet (or number) to the end of each name.

In the example below, I split a file containing 100,000 lines. I instruct split to use numeric suffixes (-d), put 10,000 lines in each split file (-l 10000) and use suffixes of length 3 (-a 3). As a result, ten split files are created, each with 10,000 lines.

$ ls
hugefile

$ wc -l hugefile
100000 hugefile

$ split -d -l 10000 -a 3 hugefile hugefile.split.

$ ls
hugefile                hugefile.split.005
hugefile.split.000      hugefile.split.006
hugefile.split.001      hugefile.split.007  
hugefile.split.002      hugefile.split.008
hugefile.split.003      hugefile.split.009
hugefile.split.004

$ wc -l *split*
 10000 hugefile.split.000
 10000 hugefile.split.001
 10000 hugefile.split.002
 10000 hugefile.split.003
 10000 hugefile.split.004
 10000 hugefile.split.005
 10000 hugefile.split.006
 10000 hugefile.split.007
 10000 hugefile.split.008
 10000 hugefile.split.009
100000 total

Sunday, October 02, 2011

Better Bash Completion for Tmux

In my previous post, I wrote about how awesome tmux is for managing multiple terminals. However, even though it is widely used, I haven't been able to find a good Bash completion script for it. The tmux package does come with bash_completion_tmux.sh but this does not complete command options or command aliases. So I wrote a better version which completes tmux commands, aliases and their options. However, there is still room for improvement. It would be nice if it could complete session and window names too, but I haven't found the time to implement this yet.

Here is a demo:

$ tmux lis[TAB]
list-buffers   list-clients   list-commands
list-keys      list-panes     list-sessions  list-windows

$ tmux list-windows -[TAB]
-a -t

$ tmux list-windows -a
sharfah:0: less [180x82] [layout f0de,180x82,0,0]
sharfah:1: tmp [180x82] [layout f0de,180x82,0,0] (active)
sharfah:2: isengard [180x82] [layout f0de,180x82,0,0]
sharfah:3: java [180x82] [layout f0de,180x82,0,0]
My completion script is shown below. You need to source it in your Bash profile. Alternatively, save it to your Bash completion directory e.g. ~/.bash/.bash/.bash_completion.d and it should automatically get picked up.

The script is also available in my GitHub dotfiles repository. If you can improve it, fork it and send me a pull request!

#
# tmux completion
# by Fahd Shariff
#
_tmux() {
  # an array of commands and their options
  declare -A tmux_cmd_map
  tmux_cmd_map=( ["attach-session"]="-dr -t target-session" \
                 ["bind-key"]="-cnr -t key-table key command arguments" \
                 ["break-pane"]="-d -t target-pane" \
                 ["capture-pane"]="-b buffer-index -E end-line -S start-line -t target-pane" \
                 ["choose-buffer"]="-t target-window template" \
                 ["choose-client"]="-t target-window template" \
                 ["choose-session"]="-t target-window template" \
                 ["choose-window"]="-t target-window template" \
                 ["clear-history"]="-t target-pane" \
                 ["clock-mode"]="-t target-pane" \
                 ["command-prompt"]="-I inputs -p prompts -t target-client template" \
                 ["confirm-before"]="-p prompt -t target-client command" \
                 ["copy-mode"]="-u -t target-pane" \
                 ["delete-buffer"]="-b buffer-index" \
                 ["detach-client"]="-P -s target-session -t target-client" \
                 ["display-message"]="-p -c target-client -t target-pane message" \
                 ["display-panes"]="-t target-client" \
                 ["find-window"]="-t target-window match-string" \
                 ["has-session"]="-t target-session" \
                 ["if-shell"]="shell-command command" \
                 ["join-pane"]="-dhv -p percentage|-l size -s src-pane -t dst-pane" \
                 ["kill-pane"]="-a -t target-pane" \
                 ["kill-server"]="kill-server" \
                 ["kill-session"]="-t target-session" \
                 ["kill-window"]="-t target-window" \
                 ["last-pane"]="-t target-window" \
                 ["last-window"]="-t target-session" \
                 ["link-window"]="-dk -s src-window -t dst-window" \
                 ["list-buffers"]="list-buffers" \
                 ["list-clients"]="-t target-session" \
                 ["list-commands"]="list-commands" \
                 ["list-keys"]="-t key-table" \
                 ["list-panes"]="-as -t target" \
                 ["list-sessions"]="list-sessions" \
                 ["list-windows"]="-a -t target-session" \
                 ["load-buffer"]="-b buffer-index path" \
                 ["lock-client"]="-t target-client" \
                 ["lock-server"]="lock-server" \
                 ["lock-session"]="-t target-session" \
                 ["move-window"]="-dk -s src-window -t dst-window" \
                 ["new-session"]="-d -n window-name -s session-name -t target-session -x width -y height command" \
                 ["new-window"]="-adk -n window-name -t target-window command" \
                 ["next-layout"]="-t target-window" \
                 ["next-window"]="-a -t target-session" \
                 ["paste-buffer"]="-dr -s separator -b buffer-index -t target-pane" \
                 ["pipe-pane"]="-t target-pane-o command" \
                 ["previous-layout"]="-t target-window" \
                 ["previous-window"]="-a -t target-session" \
                 ["refresh-client"]="-t target-client" \
                 ["rename-session"]="-t target-session new-name" \
                 ["rename-window"]="-t target-window new-name" \
                 ["resize-pane"]="-DLRU -t target-pane adjustment" \
                 ["respawn-pane"]="-k -t target-pane command" \
                 ["respawn-window"]="-k -t target-window command" \
                 ["rotate-window"]="-DU -t target-window" \
                 ["run-shell"]="command" \
                 ["save-buffer"]="-a -b buffer-index" \
                 ["select-layout"]="-np -t target-window layout-name" \
                 ["select-pane"]="-lDLRU -t target-pane" \
                 ["select-window"]="-lnp -t target-window" \
                 ["send-keys"]="-t target-pane key " \
                 ["send-prefix"]="-t target-pane" \
                 ["server-info"]="server-info" \
                 ["set-buffer"]="-b buffer-index data" \
                 ["set-environment"]="-gru -t target-session name value" \
                 ["set-option"]="-agsuw -t target-session|target-window option value" \
                 ["set-window-option"]="-agu -t target-window option value" \
                 ["show-buffer"]="-b buffer-index" \
                 ["show-environment"]="-g -t target-session" \
                 ["show-messages"]="-t target-client" \
                 ["show-options"]="-gsw -t target-session|target-window" \
                 ["show-window-options"]="-g -t target-window" \
                 ["source-file"]="path" \
                 ["split-window"]="-dhvP -p percentage|-l size -t target-pane command" \
                 ["start-server"]="start-server" \
                 ["suspend-client"]="-t target-client" \
                 ["swap-pane"]="-dDU -s src-pane -t dst-pane" \
                 ["swap-window"]="-d -s src-window -t dst-window" \
                 ["switch-client"]="-lnp -c target-client -t target-session" \
                 ["unbind-key"]="-acn -t key-table key" \
                 ["unlink-window"]="-k -t target-window" )

   declare -A tmux_alias_map
   tmux_alias_map=( ["attach"]="attach-session" \
                  ["detach"]="detach-client" \
                  ["has"]="has-session" \
                  ["lsc"]="list-clients" \
                  ["lscm"]="list-commands" \
                  ["ls"]="list-sessions" \
                  ["lockc"]="lock-client" \
                  ["locks"]="lock-session" \
                  ["new"]="new-session" \
                  ["refresh"]="refresh-client" \
                  ["rename"]="rename-session" \
                  ["showmsgs"]="show-messages" \
                  ["source"]="source-file" \
                  ["start"]="start-server" \
                  ["suspendc"]="suspend-client" \
                  ["switchc"]="switch-client" \
                  ["breakp"]="break-pane" \
                  ["capturep"]="target-pane]" \
                  ["displayp"]="display-panes" \
                  ["findw"]="find-window" \
                  ["joinp"]="join-pane" \
                  ["killp"]="kill-pane" \
                  ["killw"]="kill-window" \
                  ["lastp"]="last-pane" \
                  ["last"]="last-window" \
                  ["linkw"]="link-window" \
                  ["lsp"]="list-panes" \
                  ["lsw"]="list-windows" \
                  ["movew"]="move-window" \
                  ["neww"]="new-window" \
                  ["nextl"]="next-layout" \
                  ["next"]="next-window" \
                  ["pipep"]="pipe-pane" \
                  ["prevl"]="previous-layout" \
                  ["prev"]="previous-window" \
                  ["renamew"]="rename-window" \
                  ["resizep"]="resize-pane" \
                  ["respawnp"]="respawn-pane" \
                  ["respawnw"]="respawn-window" \
                  ["rotatew"]="rotate-window" \
                  ["selectl"]="select-layout" \
                  ["selectp"]="select-pane" \
                  ["selectw"]="select-window" \
                  ["splitw"]="[shell-command]" \
                  ["swapp"]="swap-pane" \
                  ["swapw"]="swap-window" \
                  ["unlinkw"]="unlink-window" \
                  ["bind"]="bind-key" \
                  ["lsk"]="list-keys" \
                  ["send"]="send-keys" \
                  ["unbind"]="unbind-key" \
                  ["set"]="set-option" \
                  ["setw"]="set-window-option" \
                  ["show"]="show-options" \
                  ["showw"]="show-window-options" \
                  ["setenv"]="set-environment" \
                  ["showenv"]="show-environment" \
                  ["confirm"]="confirm-before" \
                  ["display"]="display-message" \
                  ["clearhist"]="clear-history" \
                  ["deleteb"]="delete-buffer" \
                  ["lsb"]="list-buffers" \
                  ["loadb"]="load-buffer" \
                  ["pasteb"]="paste-buffer" \
                  ["saveb"]="save-buffer" \
                  ["setb"]="set-buffer" \
                  ["showb"]="show-buffer" \
                  ["if"]="if-shell" \
                  ["lock"]="lock-server" \
                  ["run"]="run-shell" \
                  ["info"]="server-info" )

   local cur="${COMP_WORDS[COMP_CWORD]}"
   local prev="${COMP_WORDS[COMP_CWORD-1]}"
   COMPREPLY=()

   # completing an option
   if [[ "$cur" == -* ]]; then
     #tmux options
     if [[ "$prev" == "tmux" ]]; then
         COMPREPLY=( $( compgen -W "-2 -8 -c -f -L -l -q -S -u -v -V" -- $cur ) )
     else
         #find the tmux command so that we can complete the options
         local cmd="$prev"
         local i=$COMP_CWORD
         while [[ "$cmd" == -* ]]
         do
             cmd="${COMP_WORDS[i]}"
             ((i--))
         done

         #if it is an alias, look up what the alias maps to
         local alias_cmd=${tmux_alias_map[$cmd]}
         if [[ -n ${alias_cmd} ]]
         then
             cmd=${alias_cmd}
         fi

         #now work out the options to this command
         local opts=""
         for opt in ${tmux_cmd_map[$cmd]}
         do
              if [[ "$opt" == -* ]]; then
                  len=${#opt}
                  i=1
                  while [ $i -lt $len ]; do
                      opts="$opts -${opt:$i:1}"
                      ((i++))
                  done
              fi
         done
         COMPREPLY=($(compgen -W "$opts" -- ${cur}))
     fi
   else
     COMPREPLY=($(compgen -W "$(echo ${!tmux_cmd_map[@]} ${!tmux_alias_map[@]})" -- ${cur}))
   fi
   return 0
}
complete -F _tmux tmux
Related posts:
Managing Multiple Terminals with Tmux Writing your own Bash Completion Function

Saturday, October 01, 2011

Managing Multiple Terminals with Tmux

I've started using tmux, which is a "terminal multiplexer", similar to screen. It allows you to manage a number of terminals from a single screen. So, for example, instead of having 5 PuTTY windows cluttering up your desktop, you now have only one window, containing 5 terminals. If you close this window, you can simply open a new one and "attach" to your running tmux session, to get all your terminals back at the same state you left them in.

There are lots of cool things you can do with tmux. For example, you can split a terminal window horizontally or vertically into "panes". This allows you to look at files side by side, or simply watch a process in one pane while you do something else in another.

I took the following screenshot of tmux in action:


The status bar along the bottom shows that I have 5 terminal windows open. I am currently in the one labelled "1-demo" and within this window I have 4 panes, each running a different command.

There are quite a few key bindings to learn, but once you have mastered them you will be able to jump back and forth between windows, move them around and kill them without lifting your hands off the keyboard. You can also set your own key bindings for things you do frequently. For example, my Ctrl-b / binding splits my window vertically and opens up a specified man page on the right. My Ctrl+b S binding allows me to SSH to a server in a new window.

Here is my tmux configuration taken from ~/.tmux.conf which shows my key bindings and colour setup. You can download this file from my GitHub dotfiles repository.

bind | split-window -h
bind - split-window -v
bind _ split-window -v
bind R source-file ~/.tmux.conf \; display-message "tmux.conf reloaded!"

bind / command-prompt -p "man" "split-window -h 'man %%'"
bind S command-prompt -p "ssh" "new-window -n %1 'exec ssh %1'"
bind h split-window -h  "man tmux"

set -g terminal-overrides 'xterm*:smcup@:rmcup@'

set -g history-limit 9999

# Terminal emulator window title
set -g set-titles on
set -g set-titles-string '#S:#I.#P #W'

# notifications
setw -g monitor-activity on
setw -g visual-activity on

# auto rename
setw -g automatic-rename on

# Clock
setw -g clock-mode-colour green
setw -g clock-mode-style 24

# Window status colors
setw -g window-status-bg colour235
setw -g window-status-fg colour248
setw -g window-status-alert-attr underscore
setw -g window-status-alert-bg colour235
setw -g window-status-alert-fg colour248
setw -g window-status-current-attr bright
setw -g window-status-current-bg colour235
setw -g window-status-current-fg colour248

# Message/command input colors
set -g message-bg colour240
set -g message-fg yellow
set -g message-attr bright

# Status Bar
set -g status-bg colour235
set -g status-fg colour248
set -g status-interval 1
set -g status-left '[#H]'
set -g status-right ''

set -g pane-border-fg white
set -g pane-border-bg default
set -g pane-active-border-fg white
set -g pane-active-border-bg default