Technetra

Archive for November, 2008

Pairing Stubborn Bluetooth Devices in Fedora 10 & Ubuntu Ibex

Saturday, November 22nd, 2008

If you’re having problems pairing Bluetooth devices with the latest BlueZ 4 GNOME software (bluez-gnome-1.8) try this script. This script offers a work-around for the problem of pairing Bluetooth audio devices like Sony’s SRS-BT100 with the latest BlueZ GNOME wizard. The following script forces the BlueZ 4 audio service to request a pincode from the user for a particular device. In the case of SRS-BT100, the pincode is hardcoded as ‘0000′. So just enter your device’s pincode when prompted by the GNOME Bluetooth applet and then start playing your favorite audio files. Note: this script adds an ALSA virtual device named ‘bt_audio’ to the user’s .asoundrc ALSA configuration file.

For more information on BlueZ, please consult www.bluez.org and wiki.bluez.org.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
#!/bin/bash
#
# Copyright (c) 2008 Technetra Corp
# 
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation
# files (the "Software"), to deal in the Software without
# restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following
# conditions:
# 
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
#
echo "Please place your Bluetooth audio device into pairing mode, then press <Enter>"
read
echo "Scanning for Bluetooth audio device..."
scan=`hcitool scan | sed 's/^\s*//' | grep -v 'Scanning'`
line=$scan
if [[ "`echo "$scan" | wc -l`" -gt "1" ]]; then
  echo "select device:"
  save_ifs=$IFS
  IFS="
"
  select line in $scan; do
    #echo got $line
    break
  done
  IFS=$save_ifs
fi
BT_BDADDR=`echo $line | awk '{print $1}'`
[[ -z "$BT_BDADDR" ]] && echo no bluetooth device found && exit
echo "Selected bluetooth device: $BT_BDADDR"
# => e.g., 00:11:22:33:44:55
 
echo "Writing new ~.asoundrc..."
cat >> ~/.asoundrc <<EOD
pcm.bt_audioraw {
        type bluetooth
        device $BT_BDADDR
        profile "auto"
}
pcm.bt_audio {
	type plug
        slave.pcm "bt_audioraw"
        hint {
            show on
            description "Bluetooth audio device"
        }
}
EOD
 
echo "Getting default adapter..."
_BT_ADAPTER=`dbus-send --system --print-reply --dest=org.bluez / \
    org.bluez.Manager.DefaultAdapter|awk '/object path/ {print $3}'`
BT_ADAPTER=${_BT_ADAPTER//\"/}
echo "$BT_ADAPTER"
 
echo "Removing any stale Bluetooth audio device:"
_OLD_BT_DEVICE=`dbus-send --system --print-reply --dest=org.bluez $BT_ADAPTER \
    org.bluez.Adapter.FindDevice string:$BT_BDADDR|awk '/object path/ {print $3}'`
OLD_BT_DEVICE=${_OLD_BT_DEVICE//\"/}
dbus-send --system --print-reply --dest=org.bluez $BT_ADAPTER \
    org.bluez.Adapter.RemoveDevice objpath:$OLD_BT_DEVICE
 
echo "Creating Bluetooth audio device:"
_BT_DEVICE=`dbus-send --system --print-reply --dest=org.bluez $BT_ADAPTER \
    org.bluez.Adapter.CreateDevice string:$BT_BDADDR|awk '/object path/ {print $3}'`
BT_DEVICE=${_BT_DEVICE//\"/}
echo "$BT_DEVICE"
 
# optional: echo "Connecting -- BlueZ applet will prompt for pin..."
# optional: dbus-send --system --print-reply --dest=org.bluez \
# optional:     $BT_DEVICE org.bluez.AudioSink.Connect
#
# NOTE: if the above dbus-send is NOT executed, then a pairing
# request for an A2DP device will be initiated the first time
# the ALSA virtual device bt_audio is used. Otherwise, when
# "AudioSink.Connect" is executed, a pairing request to the device
# will be initiated immediately and then ALSA will subsequently use
# this paired connection for playing sound through bt_audio. Executing
# "AudioSink.Connect" is useful for preparing devices ahead of time
# to avoid undesired delays and to avoid pairing mode timeouts.
# 
# Also note that a device conforming to the headset profile (HSP)
# does not support "AudioSink.Connect". However, this script (without
# running "AudioSink.Connect") will still work because BlueZ will
# request pairing when "Adapter.CreateDevice" is executed for HSP
# devices. You can check out your device's specific capabilities
# using sdptool.
 
#echo "Test sound sample..."
#aplay -D bt_audio your_sound_file.wav

CodeMagic: rename.rb - a Ruby version of Perl’s rename command

Tuesday, November 11th, 2008

rename.rb renames multiple files. The rename.rb script is a Ruby version of the popular Perl command found in many modern Linux distributions.

rename.rb [ -v ] [ -n] [ -f ] rubyexpr [ files ]

rename.rb was adapted from Larry Wall and Robin Barker’s similar Perl script. This version, however, supports Ruby-style instead of Perl-style filename transformations. Of particular interest in the Ruby script is the fact that an internal variable f, representing the current filename being transformed, is exposed on the external command line to the caller. This leads to the potential application of interesting Ruby-based side effects involving passive or active manipulation of the filename or the contents of the file while it is being renamed.

For example, clever if somewhat silly transformations of filenames can be accomplished. You can turn all the filenames in a directory into their natural palindromes with the command:

./rename.rb -n 'concat(f.chop.reverse)' *

Or prefix each filename with a “graphical” representation of the corresponding file size.

./rename.rb -n 'sub(/^/,%(#)*Math.log(3+File.size(f)).floor+%(-))' *

In these commands, the -n option tells rename not to actually perform the rename, but only to show what would have been done. The commands

'concat(f.chop.reverse)'
'sub(/^/,%(#)*Math.log(3+File.size(f)).floor+%(-))'

are enclosed in quotes (due to the embedded parentheses and other meta-characters) and applied to each filename in turn. Finally, the files to be renamed comprise the remaining arguments on the command line. In this case the shell glob expression * tells shell to produce a list of all file and directory names in the current directory and provide this list as arguments to rename.rb.

The following command adds the md5sum of the contents of the file to the new name of the corresponding file:

./rename.rb -n 'concat("."+`md5sum #{f}|cut -d" " -f 1`).chomp' *

Lower-casing each file or directory name in a directory is as simple as:

./rename.rb -n downcase *

Like Perl’s version, this script will not rename a file if the transformed filename is the same as the original filename unless --force is requested. Instead, the program will silently resume processing any remaining filenames.

The options for the Ruby script are:

-v, --verbose
Print names of files successfully renamed
-n, --no-act
Show which files would have been renamed, but do not actually perform the action
-f, --force
Overwrite existing files

If filenames are not provided, rename.rb will read names from standard input.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
#!/usr/bin/env ruby
#
# rename.rb is a Ruby version of the popular Perl command found in many
# modern Linux distributions.
#
# rename.rb was adapted from Larry Wall and Robin Barker's similar Perl
# script. This version, however, supports Ruby- instead of Perl-style
# filename transformations. Of particular interest in the Ruby script
# is the fact that an internal variable 'f', representing the current
# filename being renamed, is exposed on the external command line to
# the caller. This leads to the potential application of interesting
# Ruby-based side effects involving passive or active manipulation of
# the filename or the contents of the file while it is being renamed.
#
# For example, clever if somewhat silly transformations of filenames
# can be accomplished.  You can turn all the files in a directory into
# their natural palindromes with the command:
#
#   ./rename.rb -n 'concat(f.chop.reverse)' *
#
# In this command, the -n option tells rename not to actually perform
# the rename, but only to show what would have been done. The command
# 'concat(f.chop.reverse)' is enclosed in quotes (due to the embedded
# parentheses) and applied to each filename in turn. Finally, the
# filenames to be renamed comprise the remaining arguments on the command
# line. In this case the shell glob expression * tells shell to produce
# a list of all the file and directory names in the current directory
# and provide this list as arguments to rename.rb.
#
# The following command adds the md5sum of the contents of the file to
# the new name of the corresponding file:
#
#   ./rename.rb -n 'concat("."+`md5sum #{f}|cut -d" " -f 1`).chomp' *
#
# Lower-casing each file or directory name in a directory is accomplished
# by this simple command:
#
#   ./rename.rb -n downcase *
#
# Like Perl's version, this script will not rename a file if the
# transformed filename is the same as the original filename unless
# --force is requested. Instead, the program will silently resume
# processing any remaining filenames.
#
# The options for rename.rb:
#
#     -v, --verbose
#          Print names of files successfully renamed
#     -n, --no-act
#          Show which files would have been renamed, but do not actually perform the action
#     -f, --force
#          Overwrite existing files
#
# If filenames are not provided, rename.rb will read names from standard input.
#
# Copyright (c) 2008 Technetra Corp
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation
# files (the "Software"), to deal in the Software without
# restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following
# conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
#
require 'optparse'
 
verbose = no_act = force = op = nil
OptionParser.new do |opts|
  opts.banner = "Usage: rename.rb [options]"
  opts.on("-v", "--verbose", "Run verbosely") { |v| verbose = v }
  opts.on("-n", "--[no-]no-act", "Show what files would have been renamed") { |n| no_act = n }
  opts.on("-f", "--force", "Force, overwrite existing files") { |f| force = f }
end.parse!
 
verbose = true if no_act
op = ARGV.shift
 
if (argvcopy = ARGV).empty?
    print "reading filenames from STDIN\n" if verbose
    argvcopy = STDIN.read.split("\n")
end
 
argvcopy.each { |f|
    was = f
    f = eval("f.dup.#{op}")
    next if was == f # ignore quietly
    if File.exist?(f) and !force
	print  "#{was} not renamed: #{f} already exists\n"
    elsif no_act or File.rename(was, f)
	print "#{was} renamed as #{f}\n" if verbose
    else
	print "Can't rename #{was} #{f}: $!\n"
    end
}

CodeMagic: serialize_with_cache - an approach to serializing and de-serializing arbitrary data in bash using an MRU cache

Sunday, November 9th, 2008

The program serialize_with_cache demonstrates an approach to serializing and de-serializing arbitrary data in bash. This version explores the use of a cache to accelerate conversion of characters in the serialization string. The cache improves serialization performance on typical text by about 25%. However, performance is still quite slow so that string serialization using native bash mechanisms remains infeasible for anything but small datasets.

After making it executable, a simple way to run the script is by invoking:

./serialize_with_cache.sh

serialize_with_cache then runs a small self-test harness. The test exercises several small string serializations and de-serializations and then times a cycle of deflating and re-inflating the contents of a common system file (read only, of course) as well as measuring the performance speed-up due to cache effects.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
#!/bin/bash
#
# serialize_with_cache demonstrates an approach to serializing
# and de-serializing arbitrary data in bash.  This version explores
# the use of a cache to accelerate conversion of characters in the
# serialization string. The cache improves serialization performance
# on typical text by about 25%. However, performance is still quite
# slow so that string serialization using native bash mechanisms
# remains infeasible for anything but small datasets. 
#
# A simple way to run the script is by invoking:
#
#     ./serialize_with_cache.sh
#
# serialize_with_cache then runs a small self-test harness. The test
# exercises several small string serializations and de-serializations
# and then times a cycle of deflating and re-inflating the contents of
# a common system file (read only, of course) as well as measuring the
# performance speed-up due to cache effects.
# 
# Copyright (c) 2008 Technetra Corp
# 
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation
# files (the "Software"), to deal in the Software without
# restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following
# conditions:
# 
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
#
PASSED=0
TOTAL=0
I=1
MRU[0]="za61"
CACHE_SIZE=200
find_char_no_cache() {
  #sleep 1
  local hex_result=$1
  local in_char=$2
  local hex_digits1 hex_digits2=(0 1 2 3 4 5 6 7 8 9 A B C D E F)
  if [[ "$in_char" > $'\x6f' ]]; then
    hex_digits1=(7 8 9 A B C D E F)
  elif [[ "$in_char" > $'\x5f' ]]; then
    hex_digits1=(6)
  elif [[ "$in_char" > $'\x4f' ]]; then
    hex_digits1=(5)
  elif [[ "$in_char" > $'\x3f' ]]; then
    hex_digits1=(4)
  elif [[ "$in_char" > $'\x2f' ]]; then
    hex_digits1=(3)
  elif [[ "$in_char" > $'\x1f' ]]; then
    hex_digits1=(2)
  else
    hex_digits1=(0 1)
  fi
  local finished=0
  local c t targ hxi hxj
  for hxi in ${hex_digits1[@]}; do
    for hxj in ${hex_digits2[@]}; do
      c=$hxi$hxj
      #t="`eval echo z\$\'\\\x$c\'z`"
      t="`echo -e "z\x${c}z"`"
      targ=${t:1:1}
      if [[ "$in_char" == "$targ" ]]; then
        #echo "matched character index $c"
        finished=1
        break
      fi
    done
    [[ $finished -eq 1 ]] && break
  done
  eval $hex_result=$c
} 
find_char_with_cache() {
  #sleep 1
  local hex_result=$1
  local in_char=$2
  local j=$(((I+(CACHE_SIZE-1))%CACHE_SIZE))
  while [[ j -ne I%CACHE_SIZE ]]; do
    m="${MRU[j]}"
    if [[ "$m" ]]; then
      j=$(((j+(CACHE_SIZE-1))%CACHE_SIZE))
      if [[ "${m:1:1}" == "$in_char" ]]; then
        eval $hex_result=${m:2:2}
        #echo "got cache hit \"$in_char\""
        I=$((I%CACHE_SIZE))
        MRU[I++]="`echo -e "z$in_char${m:2:2}"`"
        return
      fi 
    else
      #echo "missed cache for \"$in_char\""
      break
    fi
    #sleep 1
  done 
  local hex_digits1 hex_digits2=(0 1 2 3 4 5 6 7 8 9 A B C D E F)
  if [[ "$in_char" > $'\x6f' ]]; then
    hex_digits1=(7 8 9 A B C D E F)
  elif [[ "$in_char" > $'\x5f' ]]; then
    hex_digits1=(6)
  elif [[ "$in_char" > $'\x4f' ]]; then
    hex_digits1=(5)
  elif [[ "$in_char" > $'\x3f' ]]; then
    hex_digits1=(4)
  elif [[ "$in_char" > $'\x2f' ]]; then
    hex_digits1=(3)
  elif [[ "$in_char" > $'\x1f' ]]; then
    hex_digits1=(2)
  else
    hex_digits1=(0 1)
  fi
  local finished=0
  local c t targ hxi hxj
  for hxi in ${hex_digits1[@]}; do
    for hxj in ${hex_digits2[@]}; do
      c=$hxi$hxj
      #t="`eval echo z\$\'\\\x$c\'z`"
      t="`echo -e "z\x${c}z"`"
      targ=${t:1:1}
      if [[ "$in_char" == "$targ" ]]; then
        #echo "matched character index $c"
        finished=1
        I=$((I%CACHE_SIZE))
        MRU[I++]="`echo -e "z$targ$c"`"
        break
      fi
    done
    [[ $finished -eq 1 ]] && break
  done
  eval $hex_result=$c
}
deflate() {
  local result=$1
  local s="$2"
  local which_find_char=$3
  local i=0 deflated out_hex
  local c="${s:i++:1}"
  while [[ "$c" ]]; do
    $which_find_char out_hex "$c"
    deflated=$deflated$out_hex
    c="${s:i++:1}"
  done 
  eval $result=$deflated
}
inflate() {
  local result=$1
  local hex_string=$2 
  local i=0 inflated t targ
  local h=${hex_string:i:2}
  i=$((i+2))
  while [[ "$h" ]]; do
    #t="`eval echo z\$\'\\\x$h\'z`"
    t="`echo -e "z\x${h}z"`"
    targ=${t:1:1}
    [[ "$targ" == '"' ]] && targ='\"'
    inflated="$inflated$targ"
    h=${hex_string:i:2}
    i=$((i+2))
  done
  eval "$result=\"$inflated\""
}
test_serialize () {
  local in_orig="$1"
  local which_find_char=${2:-find_char_with_cache}
  local stat hex copy
  deflate hex "$in_orig" $which_find_char
  inflate copy $hex
  if [[ "$in_orig" == "$copy" ]]; then
    stat=passed
    ((PASSED++))
  else
    stat=failed
  fi
  ((TOTAL++))
  echo "$stat test $TOTAL (cache=$which_find_char), hex=$hex, original=\"$in_orig\""
}
time_file () {
  echo "timing serialization for file $1..."
  test_serialize "$(cat $1)" $2
}
###
### MAIN
###
test_serialize "a\"c"
test_serialize "a\'c"
test_serialize "$(echo -e " a\n\tb")"
test_serialize "'%' in a prerequisite of a pattern rule stands for the same stem that was matched by the"
time_file /etc/passwd
echo "checking performance improvement due to cache..."
# built-in bash reserved word, time, requires execution in a subshell to capture its stderr
# output
export TIMEFORMAT=%3R
t1=`(time test_serialize "$(head -n 25 /etc/passwd)" >/dev/null) 2>&1`
echo "continuing..."
t2=`(time test_serialize "$(head -n 25 /etc/passwd)" find_char_no_cache >/dev/null) 2>&1`
# use integer arithmetic and add leading 1 to guard against misinterpretation
# as octal because of leading zeros
l1=${#t1}
l2=${#t2}
max=$((l1>l2?l1:l2))
pad=100000
pad=${pad:0:max}
z1=${pad:0:max-l1+1}${t1/./}
z2=${pad:0:max-l2+1}${t2/./}
echo "TEST SUMMARY: passed $PASSED of $TOTAL tests" \
     "(cache speed-up approx: $((((z2-z1)*100)/(z2-pad)))%)"
# end serialize_with_cache

CodeMagic: serialize - an approach to serializing and de-serializing arbitrary data entirely within bash

Sunday, November 9th, 2008

The program serialize demonstrates an approach to serializing and de-serializing arbitrary data entirely within the native facilities of bash. Performance, however, is sufficiently slow that this approach is infeasible for anything but small datasets.

After making it executable, a simple way to run the script is by invoking:

./serialize.sh

serialize then runs a small self-test harness. The test includes a timing cycle of deflating and re-inflating the contents of the file holding the script itself as well as serializing the entire ASCII sequence except for x’00′. Although performance is painfully slow, this demonstration exposes scripting techniques for sophisticated data handling including advanced array manipulation, as well as binary and hex string processing. Cache effects are explored in a second, related script named serialize_with_cache.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
#!/bin/bash
#
# serialize demonstrates an approach to serializing and de-serializing
# arbitrary data entirely within the native facilities of bash.
# Performance, however, is sufficiently slow that this approach is
# infeasible for anything but small datasets.
#
# A simple way to run the script is by invoking:
#
#     ./serialize.sh 
#
# serialize then runs a small self-test harness. The test includes a
# timing cycle of deflating and re-inflating the contents of the file
# holding the script itself as well as serializing the entire ASCII
# sequence except for x'00'. Although performance is painfully slow,
# the demonstration exposes scripting techniques for sophisticated data
# handling including advanced array manipulation, as well as binary
# and hex string processing. Cache effects are explored in a second,
# related script named serialize_with_cache.
# 
# Copyright (c) 2008 Technetra Corp
# 
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation
# files (the "Software"), to deal in the Software without
# restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following
# conditions:
# 
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
# 
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
#
PASSED=0
TOTAL=0
find_char() {
  local hex_result=$1
  local in_char=$2
  local hex_digits1 hex_digits2=(0 1 2 3 4 5 6 7 8 9 A B C D E F)
  if [[ "$in_char" > $'\x6f' ]]; then
    hex_digits1=(7 8 9 A B C D E F)
  elif [[ "$in_char" > $'\x5f' ]]; then
    hex_digits1=(6)
  elif [[ "$in_char" > $'\x4f' ]]; then
    hex_digits1=(5)
  elif [[ "$in_char" > $'\x3f' ]]; then
    hex_digits1=(4)
  elif [[ "$in_char" > $'\x2f' ]]; then
    hex_digits1=(3)
  elif [[ "$in_char" > $'\x1f' ]]; then
    hex_digits1=(2)
  else
    hex_digits1=(0 1)
  fi
  local finished=0
  local c t targ hxi hxj
  for hxi in ${hex_digits1[@]}; do
    for hxj in ${hex_digits2[@]}; do
      c=$hxi$hxj
      #t="`eval echo z\$\'\\\x$c\'z`"
      t="`echo -e "z\x${c}z"`"
      targ=${t:1:1}
      if [[ "$in_char" == "$targ" ]]; then
        finished=1
        break
      fi
    done
    [[ $finished -eq 1 ]] && break
  done
  eval $hex_result=$c
}
deflate() {
  local result=$1
  local s="$2"
  local i=0 deflated out_hex
  local c="${s:i++:1}"
  while [[ "$c" ]]; do
    find_char out_hex "$c"
    deflated=$deflated$out_hex
    c="${s:i++:1}"
  done 
  eval $result=$deflated
}
inflate() {
  local result=$1
  local hex_string=$2 
  local i=0 inflated t targ
  local h=${hex_string:i:2}
  i=$((i+2))
  while [[ "$h" ]]; do
    #t="`eval echo z\$\'\\\x$h\'z`"
    t="`echo -e "z\x${h}z"`"
    targ=${t:1:1}
    [[ "$targ" == '"' ]] && targ='\"'
    inflated="$inflated$targ"
    h=${hex_string:i:2}
    i=$((i+2))
  done
  eval "$result=\"$inflated\""
}
test_full_hex_table() {
  local hex_string=$1 
  local i=0 inflated t hex stat=failed
  local h=${hex_string:i:2}
  i=$((i+2))
  while [[ "$h" ]]; do
    t="`echo -e "z\x${h}z"`"
    inflated="$inflated${t:1:1}"
    h=${hex_string:i:2}
    i=$((i+2))
  done
  deflate hex "$inflated"
  if [[ "$hex_string" == "$hex" ]]; then
    stat=passed
    ((PASSED++))
  fi
  ((TOTAL++))
  echo -n "$stat test $TOTAL, hex=$hex, original=\"$(echo $inflated | cat -v)\"\n"
}
test_file_content() {
  local s="$1"
  local i=0 deflated out_hex inflated t hex stat=failed
  local c="${s:i++:1}"
  while [[ "$c" ]]; do
    find_char out_hex "$c"
    deflated=$deflated$out_hex
    c="${s:i++:1}"
  done 
  i=0
  local h=${deflated:i:2}
  i=$((i+2))
  while [[ "$h" ]]; do
    t="`echo -e "z\x${h}z"`"
    inflated="$inflated${t:1:1}"
    h=${deflated:i:2}
    i=$((i+2))
  done
  deflate hex "$inflated"
  if [[ "$deflated" == "$hex" ]]; then
    stat=passed
    ((PASSED++))
  fi
  ((TOTAL++))
  echo "$stat test $TOTAL, hex=$hex, original=\"$s\""
}
test_serialize () {
  local in_orig="$1"
  local stat hex copy
  deflate hex "$in_orig"
  inflate copy $hex
  if [[ "$in_orig" == "$copy" ]]; then
    stat=passed
    ((PASSED++))
  else
    stat=failed
  fi
  ((TOTAL++))
  echo "$stat test $TOTAL, hex=$hex, original=\"$in_orig\""
}
time_file () {
  local file=$1
  echo "timing serialization on file $file, may take several minutes..."
  time test_file_content "$(cat $file)"
}
time_full_hex_range () {
  local hex_digits=(0 1 2 3 4 5 6 7 8 9 A B C D E F)
  local copy hstr
  for hxi in ${hex_digits[@]}; do
    for hxj in ${hex_digits[@]}; do
      [[ $hxi == "0" && $hxj == "0" ]] || hstr=$hstr$hxi$hxj
    done
  done
  echo "timing serialization of fully populated (except NULL) ASCII table..."
  time test_full_hex_table $hstr
}
###
### MAIN
###
test_serialize "a\"c"
test_serialize "a\'c"
test_serialize "$(echo -e " a\n\tb")"
test_serialize "'%' in a prerequisite of a pattern rule stands for the same stem that was matched by the"
time_file $0
time_file /etc/passwd
time_full_hex_range
echo "TEST SUMMARY: passed $PASSED of $TOTAL tests"
# end serialize

CodeMagic: bashtail - a script version of the Unix tail command

Sunday, November 9th, 2008

The program bashtail simulates a simple version of the Unix tail command written in bash. To run the script, make sure bashtail.sh is executable, and then enter the following:

./bashtail -n NUMBER_OF_LINES_TO_SHOW SOME_INPUT_FILE

bashtail uses a circular buffer to retain the last n lines read from standard input or from a file. The buffer is written out in proper order when the input stream is finished.

This bash script demonstrates the proper use of local namespaces and it modularizes the work of the program through functional decomposition. To allow a function to pass back a return value, a technique similar to globbing in Perl is used. The function wishing to set a return value evals a variable name, provided as a string by the caller, into the caller’s symbol table. As a side effect, eval sets the value for the variable returned to the caller.

bashtail includes a small self-test harness which can be run using the -t argument. The output of bashtail is compared to that of tail and a short report of the tests passed is produced.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
#!/bin/bash
#
# bashtail simulates a simple version of tail written in bash.
# A simple way to run the script is by invoking:
#
#     bashtail -n NUMBER_OF_LINES_TO_SHOW SOME_INPUT_FILE
#
# bashtail uses a circular buffer to retain the last n lines
# read from stdin or from a file. The buffer is written out in
# proper order when the input stream is finished.
#
# The script demonstrates the proper use of local namespaces
# and it modularizes the work of the program through
# functional decomposition. To allow a function to pass
# back a return value, a technique similar to globbing in
# Perl is used. The function wishing to set a return value
# evals a variable name, provided as a string by the caller,
# into the caller's symbol table. As a side effect, eval sets
# the value for the variable returned to the caller.
#
# bashtail includes a small self-test harness which can be
# run using the -t argument. The output of bashtail is
# compared to that of tail and a short report of the tests
# passed is produced.
#
# Copyright (c) 2008 Technetra Corp
#
# Permission is hereby granted, free of charge, to any person
# obtaining a copy of this software and associated documentation
# files (the "Software"), to deal in the Software without
# restriction, including without limitation the rights to use,
# copy, modify, merge, publish, distribute, sublicense, and/or sell
# copies of the Software, and to permit persons to whom the
# Software is furnished to do so, subject to the following
# conditions:
#
# The above copyright notice and this permission notice shall be
# included in all copies or substantial portions of the Software.
#
# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
# EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
# OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
# NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
# HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
# WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
# FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
# OTHER DEALINGS IN THE SOFTWARE.
#
RB_ITER=
_read_lines() {
  local rbc=$1
  local N=$2
  local i=0
  local line_count=0
  local oldIFS="$IFS"; IFS=
  while read l; do
    line_count=$((line_count+1))
    i=$((i%N))
    ary[i++]="$l"
  done
  IFS="$oldIFS"
  RB_ITER=$i
  eval "$rbc=$line_count"
}
 
_print_buf() {
  local line_count=$1
  local N=$2
  local j=0
  # $RB_ITER points to start of ring buffer when line_count >= N
  # else to one past end of ring buffer when line_count < N
  if [[ line_count -lt N ]]; then
    RB_ITER=0
    N=$line_count
  fi
  while [[ j++ -lt N ]]; do
    echo "${ary[RB_ITER++%N]}"
  done
}
 
usage() {
  echo "SYNOPSIS:"
  echo -e "\t$0 [OPTION] [FILE]"
  echo "DESCRIPTION:"
  echo -e "\tOutput last part of file FILE. "
  echo -e "\tWhen no FILE is given or when FILE is -, read standard input"
  echo -en "\tExample: echo -e "
  echo "\"a\\nb\\nc\\n\\nd\"|$0 -n 3"
  echo "OPTIONS:"
  echo -e "\t-h\tThis help message"
  echo -e "\t-n N\tOutput last N lines"
  echo -e "\t-t\tTest against real tail using above example while"
  echo -e "\t\tvarying n from 6 down to 0"
  exit
}
 
test_script() {
  local passed=0
  local total=0
  local TAIL=/usr/bin/tail
  local result
  local target
  echo "testing tabs, spaces and newlines from standard input with simple echo..."
  for i in $(seq 6 -1 0); do
    total=$((total+1))
    result=$(echo -e " a\\n\\tb\\nc\\n\\nd"|$0 -n $i)
    target=$(echo -e " a\\n\\tb\\nc\\n\\nd"|$TAIL -n $i)
    if [[ "$result" == "$target" ]]; then
      passed=$((passed+1))
    else
      echo "failed simple echo test $i \"echo -e " a\\n\\tb\\nc\\n\\nd"|$0 -n $i\""
    fi
  done
  echo "testing file input..."
  for i in $(seq 10 -1 0); do
    total=$((total+1))
    result=$($0 -n $i /etc/passwd)
    target=$($TAIL -n $i /etc/passwd)
    if [[ "$result" == "$target" ]]; then
      passed=$((passed+1))
    else
      echo "failed file input test $i"
    fi
  done
  echo "testing standard input from file..."
  for i in $(seq 10 -1 0); do
    total=$((total+1))
    result=$(cat /etc/passwd|$0 -n $i )
    target=$(cat /etc/passwd|$TAIL -n $i )
    if [[ "$result" == "$target" ]]; then
      passed=$((passed+1))
    else
      echo "failed file input from stdin, test $i"
    fi
  done
  echo "passed $passed out of $total tests"
  exit
}
 
get_args() {
  local N_RET=$1; shift
  local FN_RET=$1; shift
  local arg=$1
  local N
  local FN
  case "$arg" in
    -h) usage;;
    -n) shift; N=$1; shift; FN=$1;;
    -t) test_script;;
  esac
  [[ -z "$N" ]] &amp;&amp; usage
  [[ $N -lt 1 ]] &amp;&amp; exit
  eval "$N_RET=$N; $FN_RET=$FN"
}
 
read_lines() {
  local result=$1
  local rb_count
  if [ "$FILE_NAME" -a "$FILE_NAME" != "-" ]; then
    _read_lines rb_count $HOW_MANY_LINES &lt; $FILE_NAME
  else
    _read_lines rb_count $HOW_MANY_LINES
  fi
  eval "$result=$rb_count"
}
 
print_buf() {
  local rb_count=$1
  _print_buf $rb_count $HOW_MANY_LINES
}
 
#
# MAIN
#
get_args HOW_MANY_LINES FILE_NAME $@
read_lines count
print_buf $count

© 2000-2010 Technetra. All rights reserved. Contact | Terms of Use

WordPress