Digitize all those binders full of notes

Ever find yourself referring to your old notes and school work?  Why not cram that giant stack of sketchy, error-riddled paper into a pdf or nine?  Just think of all the advantages!

• they would be more portable - you could put them on a usb stick or a phone
• they would be easier to reference while doing design work on the computer
• they would take up less space and collect fewer dead bugs 
• they might not get damaged by the leaky roof or eaten by termites
• they could be shared with people who want to learn that they can't read your shitty handwriting

Maybe you could even have the prescience to approach this task before you're two years past graduation!

Just some books and binders

Over my many years wasting my life with a bad drawing habit, I've learned one thing from flatbed scanners: graphite pencil marks on paper are reflective.  This means that for particular illumination angles, scanned or photographed pencil handwriting/drawings may be washed out very effectively.  Long ago, I simply abandoned scanners for drawing purposes because of this.  I decided to simply use my second-hand smartphone as a camera.  I grabbed some scrap metal from the pile next to the bandsaw, and after some sawing and hammering and hot glue, I created a tripod mount for the phone.  I set up a light tent and made a staging easel out of a pizza box.  After ten minutes of page flipping, I have a hundred pictures of upside-down pages that I can hardly read with my D-grade eyeballs. 

Rectification and contrast enhancement

Now, how to process the images?  For most purposes, processing of scanned books or handwriting can be done with a simple procedure:

• desaturate original image
• make a copy
• blur the copy with a large kernel
• blend the blurred mask with the original in a Divide mode
• re-adjust levels as appropriate

This works well and produces a high-contrast image.  Keep in mind though why it works.  This is essentially a high-pass filter method.  Only the low spatial frequency content survives the blurring operation and is removed in the blend operation.  This removes the effects of uneven lighting or slight paper contours, but if the page content is not restricted to narrow lines, we'll run into problems. 

Excess filtering on pages with graphics

Let's say some of the pages have printed figures or tables; the removal of low-frequency content will tend to leave only edges of any solid dark regions.  In the binders that contained occasional printed graphics, I used a different method for processing printed pages.  Since most of my handwritten notes are on yellow paper, I simply processed non-yellow pages differently.  If I know there are no graphics, I can just ignore the testing routine.

The color-testing routine finds the average color of an annulus of the page so as to ignore content and page placement inaccuracy.  One convenience of this is that images that are processed with the high-pass filter method can be re-colorized if desired.  I personally don't find this to help with visual contrast, so I didn't use it.

#!/bin/bash
# process photos of coursework pages from binders
# uses slower method of contrast mask generation and overlay 

#581 1773x2283+45+543 180
#487 1758x2286+54+546 180
#488 2220x1716+321+36 270
#221 1785x2325+51+531 180
#GDn 1755x2394+24+657 180
#471 1803x2319+33+540 180
#537 1779x2286+45+552 180

pdfname="ECE537_Integrated_Photonics"
cropsize="1779x2286+45+552" #the rect parameters for cropping
rotateamt="180"    #how much to rotate after cropping

indir="originals" #this is the local directory name where the original images are
outdir="output"   #this is the local directory name where the script will shit out the processed pages
outqual=85   #percent jpeg quality
hconly=1   #always assume pages are handwritten (used when there aren't any printed graphics pages)
retint=0   #percent retint for yellow pages
retintmode="multiply"

# ###########################################################################
if [ $hconly == 1 ]; then 
 echo "high contrast mode"
else
 echo "auto contrast mode"
fi

page=1
for infile in $(ls -1 $indir/*.jpg); do
 outfile="$outdir/output-$(printf "%04d" $page).jpg"
 jpegtran -crop $cropsize -trim -copy none $infile | \
 jpegtran -rotate $rotateamt -trim -outfile temp.jpg
 
 if [ $hconly == 0 ]; then 
  # get average page color excluding border and content
  imgstring=$(convert \( temp.jpg -threshold -1 -scale 95% \) \
    \( temp.jpg -threshold 100% -scale 80% \) \
   -gravity center -compose multiply -composite - | \
   convert temp.jpg - -alpha off -gravity center -compose copy_opacity -composite -resize 1x1 txt:)
  RGB=$(echo $imgstring | sed 's/ //g' | sed 's/(/ /g' | sed 's/)/ /g' | sed 's/,/ /g' | cut -d ' ' -f 6-8)
  R=$(echo $RGB | cut -d ' ' -f 1)
  G=$(echo $RGB | cut -d ' ' -f 2)
  B=$(echo $RGB | cut -d ' ' -f 3)
  isyel=$(echo "($R+$G)/2 > $B*1.3" | bc)
  #echo $imgstring
  echo $R $G $B ">> $page is yellow? >>" $isyel
 fi

 if [ $hconly == 1 ] || [ $isyel == 1 ]; then
  # if page is yellow, do 100% contrast enhancement and partial page re-tinting 
  if [ $retint != 0 ]; then 
   convert -modulate 100,0 temp.jpg - | \
   convert - \( +clone -filter Gaussian -resize 25% -define filter:sigma=25 -resize 400% \) -compose Divide_Src -composite - | \
   convert -level 70%,100% -quality 100 - temp.jpg
   convert temp.jpg \( +clone -background "rgb($R,$G,$B)" -compose Dst -flatten \) -compose $retintmode -composite - | \
   convert temp.jpg - -compose blend -define compose:args=$retint -composite - | \
   convert -quality $outqual - $outfile
  else
   convert -modulate 100,0 temp.jpg - | \
   convert - \( +clone -filter Gaussian -resize 25% -define filter:sigma=25 -resize 400% \) -compose Divide_Src -composite - | \
   convert -level 70%,100% -quality $outqual - $outfile  
  fi
 else
  # if page is not yellow, retain most color and do a 50% contrast enhancement
  convert -modulate 100,80 temp.jpg - | \
  convert - \( +clone -filter Gaussian -resize 25% -define filter:sigma=25 -resize 400% \) -compose Divide_Src -composite - | \
  convert - temp.jpg -compose blend -define compose:args=50 -composite - | \
  convert -level 25%,100% -quality $outqual - $outfile
 fi

 #echo $infile
 page=$[$page+1]
done

rm temp.jpg
convert $outdir/*.jpg $pdfname.pdf

The blurring method entails a size reduction and expansion.  This has two purposes: First, it speeds up blurs with a large kernel (in this case, by about a factor of 3); second, it helps reduce vignetting effects that would otherwise be caused by a simple "-blur 0,100" operation.  If a simple blur is used, it would help to crop the page oversize, then trim it down after contrast enhancement or after the blur itself.

Difference between simple blur and resize+blur methods

Of course you can guess that I'd do this in bash.  This is about as ad-hoc as they come.  It's not even externally parameterized.  I was thinking about doing this with Matlab, but I decided that I'd rather pull my hair out trying to do image stacks in ImageMagick.  Tell it where the files are, how they should be checked and compressed, and the script will grind them into a pdf for you.  I highly doubt anyone would ever actually use this ugly code, but I'm posting it anyway because I have nothing else to do. Still, I'm not dumb enough to post my horrible notes in whole.

Make sorting media files marginally easier

If you're like me, you have a hard drive with little scattered oxbow directories filled with the varied sediment of entertainment media strata.  These dark swamplands occasionally need draining and the value of each soggy log and bog frog within must be appraised.  The trick is to minimize the time required to hold one's breath.

The dominant populations one may find in such a data-wallow may include:

• product pdf's and datasheets
• books and reference documents
• videos of things exploding into flames
• good old-fashioned porn
• music videos from YouTube
• bad old-fashioned porn 
• esoteric music files from atypical sources
• graphs documenting the aggrandizement of a veiled fascism
• cat pictures

Whether it's by lack of self-discipline or because you have a good excuse, these things tend to collect and need occasional sorting and cleanup.   Some things like video and audio tend to be tedious to sort, not only because the files take time to review, but often because their sources leave them poorly named.

Recently I felt the need to abuse myself by sorting a folder full of shit video files I wasn't in the mood to watch.  It so happened that they were mostly rips from various flash players and all had meaningless hash salads for filenames.  Because of this, manually picking through each one becomes tedious.  Tab-completion is defeated; it's difficult to even remember which filename you're trying to target.  I put together a crude script to automate the review, renaming, and categorization of the ~200 files.

#!/bin/bash

SAVEIFS=$IFS
IFS=$(echo -en "\n\b")

if [ ! -d "funnyshit" ]; then mkdir funnyshit; fi
if [ ! -d "depressingshit" ]; then mkdir depressingshit; fi
if [ ! -d "infuriatingshit" ]; then mkdir infuriatingshit; fi
if [ ! -d "junk" ]; then mkdir junk; fi

for file in $(ls -1); do 
    echo -e "\033[0;36mPlaying: $file\033[0m " 
    mplayer $file --really-quiet
    sleep 1

    whiptail --inputbox "Filename Editor\nESC to skip this file" 8 60 "$file" 2>tname
    if [ $? != 0 ]; then 
        rm tname 
        echo -e "\033[0;36mSkipping $file\033[0m" 
        continue
    fi 
    newname=$(cat tname); rm tname

    if [ $newname != $file ]; then 
        mv $file $newname
        echo -e "\033[0;36mChanged $file to $newname\033[0m" 
        file=$newname
    else
        echo -e "\033[0;36mDid not change file name\033[0m"
    fi

    whiptail --radiolist "where to put it" 15 50 8 "funnyshit" "" on "depressingshit" "" off "infuriatingshit" "" off "junk" "" off 2>tdir
    if [ $? != 0 ]; then rm tdir; break; fi
    destination=$(cat tdir); rm tdir

    echo -e "\033[0;36mMoving $file to $destination\033[0m"
    mv $file $destination/
done

IFS=$SAVEIFS

The script just blindly picks up every file in the directory and plays it.  Non-video files can be manually skipped or sorted if they're identifiable.  Video files are played with Mplayer.  Once an evaluation is made, user hits escape to terminate Mplayer, and a dialog prompts for an optional new name and a choice of output directory bins.

There are a lot of ways to make this more flexible and useful in other tasks or with other filetypes, but I hope it's obvious by now that i don't really care any more.

Open documents in an application on a different X display

When running multihead with separate servers for each screen (Zaphod mode, not Xinerama), there comes occasional need to launch applications on a display other than the one currently in use.  This comes up in various tasks:

• Launch an app on screen A from a panel launcher on screen B
• Place application windows on various screens with a session recovery script
• Open a URL in a web browser from an application on a different screen

It's the third task which is the solution to a problem I encountered when setting up Thunderbird.  Unless Thunderbird is on the same X display as the browser, It will fail with the error

YOU BROKE IT, MILLIE!

I mention Thunderbird, but this could be Evolution or some other app. The target application doesn't have to be a web browser either.  In fact, if your browser and mail client are on different screens, the browser won't be able to open mailto links in the mail client either.  In these cases, the core of the solution is to temporarily change the $DISPLAY environment variable.

#!/bin/bash
# allows thunderbird to pass URLs to a browser on a different $DISPLAY

cdisp=$DISPLAY
DISPLAY=:0
palemoon $1
DISPLAY=$cdisp

Change the executable name and display numbers to suit. 

For Thunderbird specifically, enabling the use of this script requires changing the attachment handler settings.  Since there is no way to add new entries in the Attachments section of the Preferences dialog (on TB 24), open about:config and set network.protocol-handler.warn-external.http and network.protocol-handler.warn-external.https to TRUE. This forces a user dialog when a link of either type is clicked. Select the new script in both cases. Once successful, the dialog shouldn't pop up again. The resultant settings generated by these actions are visible/editable in the Attachments section of the Preferences dialog. 

It's worth noting that applying these methods to Firefox will open the URL in the existing Firefox session. If you expect to open a new Firefox window on another screen, you're pretty much shit out of luck as far as i know. There are certainly some dodgy workarounds, though.

Selectively Invert workspaces for readability

UPDATE October 2019: The script has been updated for multirow operation, better window identification, and reduced execution time.

In my recent experiences dealing with the rapid progression of cataracts and vision loss, I had to make a lot of adjustments to my computing environment to accommodate for the diffusion.  In my sight, bright regions in the field of view bleed over dark areas and obscure them extremely effectively.  The diffusion averages the brightness of the entire visual field, nearly eliminating local contrast.  This pretty much means that the thin black lines of text on a white background disappear.  White characters on a dark background bleed as well, but the impact on readability is much smaller.

OH GOD THE BRIGHTNESS

I do understand that other individuals with cataracts claim different experiences regarding contrast preference; perhaps it is the differences in the types of cataract structure that explains those things.  Regardless, my overall goal is to control bright areas of my workspace and enforce a relatively high contrast light-on-dark regime. 

This new quest covered the normal bases:

• GTK 2/3 theme
• QT4 color settings
• Desktop wallpaper
• Custom user styles for Firefox/Stylish

... but still, there are things that can't be properly themed.  Wine applications, as well as certain things like virtual machines and Matlab/Simulink all appear as horrible fuzzy bright rectangles of pain.  What the hell does one do about those?

If i could invert an individual window's colors, that would be sufficient.  The only method i can recall to do this is with Compiz, and that's not going to happen for various reasons.  Under Mint 14/XFCE, the only thing i could think to do is invert the entire X display.  Since i normally run these offending applications maximized, the amount of remaining bright area is mostly restricted to the xfce4-panel area at the top.  Inverting the entire display can be done with xcalib, but unless one wants to get blinded every time the workspace is switched, the application of the screen inversion should be automated.

#!/bin/bash

# This is an ad-hoc replacement for wm-specific workspace switching (ctrl-alt-left, ctrl-alt-right, ctrl-alt-number)
# Script conditionally inverts X display on workspaces containing offending windows
# Helps to enforce dark, hi-contrast UI despite unthemeable windows (virtual machines, wine apps, etc)
# Works best when applications are run full-screen
# directional usage: switchworkspace left|right|up|down
# explicit usage: switchworkspace workspacenumber

# this version is about 5% faster than the old 1-D version
# and does not invoke gamma enforcement and inversion checks at endpoints

# specify the workspace/pager layout
nwsx=3 # number of columns
nwsy=3 # number of rows


current=$(wmctrl -d | sed -n 's/^\([0-9]\+\) *\*.*/\1/p')

if [ $1 ]; then
 ex='^[0-9]+$'
 if ! [[ $1 =~ $ex ]]; then
  # argument is not a number
  if [ $1 == "right" ] || [ $1 == "left" ]; then
   if [ $1 == "right" ]; then
    if [ $(($current % $nwsx)) == $(($nwsx-1)) ]; then
     exit
    else
     target=$(($current+1))
    fi
   elif [ $1 == "left" ]; then
    if [ $(($current % $nwsx)) == 0 ]; then
     exit
    else
     target=$(($current-1))
    fi
   fi
  elif [ $1 == "up" ] || [ $1 == "down" ]; then
   if [ $1 == "down" ]; then
    if [ $(($current / $nwsx)) == $(($nwsy-1)) ]; then
     exit
    else
     target=$(($current+$nwsx))
    fi
   elif [ $1 == "up" ]; then
    if [ $(($current / $nwsx)) == 0 ]; then
     exit
    else
     target=$(($current-$nwsx))
    fi
   fi
  else
   echo "unknown direction"
   exit
  fi
 else
  # argument is numeric
  total=$(wmctrl -d | wc -l)
  if [ $1 -gt $total ]; then
   target=$(($total-1))
  elif [ $1 -lt 1 ]; then
   target=0
  else
   target=$(($1-1))
  fi
  
  if [ $current == $target ]; then exit; fi
 fi 
else
 echo "must specify a workspace number or direction (right/left/up/down)"
 exit
fi


# add other applications by window title keyword or window class
# if multiple windows match, all corresponding desktops will be affected
winlist=$(wmctrl -lx)
inv[0]=$(echo "$winlist" | grep "XFramePeer.com-mathworks-util-PostVMInit" | cut -d \  -f 3)
inv[1]=$(echo "$winlist" | grep "XFramePeer.MATLAB" | cut -d \  -f 3)
inv[2]=$(echo "$winlist" | grep "eagle.exe." | cut -d \  -f 3)
inv[3]=$(echo "$winlist" | grep "scad3.exe." | cut -d \  -f 3)
inv[4]=$(echo "$winlist" | grep "femm.exe." | cut -d \  -f 3)


function contains {
 case "${inv[@]}" in  *"$1"*) 
  echo 1 
  return 1 ;; 
 esac
 echo 0
}

wmctrl -s $target

A=$(contains $target)
B=$(contains $current)

#echo $current $target $A $B

if [ $A == 1 ] && [ $B != 1 ]; then

 xcalib -i -a
elif  [ $A == 1 ] && [ $B == 1 ]; then
 return
else
 xcalib -clear

 # i have gamma presets for each of my two monitors
 # issuing 'xcalib -clear' will reset the gamma to 1.00
 # so i'll need to reassert my preference depending on the active display
 # you might not need this
 # when running this from a terminal, $DISPLAY may be ":0" depending on what's happened during its session
 thisdisp=$(echo $DISPLAY)
 #echo $thisdisp
 if [ "$thisdisp" == ":0.0" ]; then
  xgamma -quiet -gamma 0.87
 else
  xgamma -quiet -gamma 1.2
 fi
fi

The script is relatively simple and a bit of effort did go into making it quick.  An array of workspace numbers is created by searching for specific WM_CLASS strings in the window list.  When switching workspaces with this script, the inversion state of the display is altered to correspond to the workspace contents.

Two workspaces: Terminator (normal), Matlab/Simulink (inverted)

Just add appropriate window title keywords and set your preferred wrapping behavior in the script.  Reassign the appropriate keybindings (Ctrl-Alt-Left, Ctrl-Alt-Right) so they execute this script instead of the inbuilt window manager functions.  Script accepts one parameter, either "prev" or "next".

Extract a document from an online reader

I'm glad i'm not going through first year undergrad general studies again.
The horseshit and scams that surround these programs never ceases to amaze me.

Someone showed me the online reader they had to use for their classes.  Though i'm sure there are others that are equally shitty, this Brytewave digibook reader from Follett was unusable.  It was extremely slow, and navigation was made difficult because it would tend to direct the user to random pages instead of the ones specified.  It was cram time at the end of the semester, and the book she's trying to study keeps closing and going to random pages.  That's how e-book technology has revolutionized education and brought us to a bright new era of virtual learning!  Fuck the rent-seeking charlatans and the government money they rode in on.

That's the extent of the rant.  How to get the fucking shitty book as a pdf:
As you can guess, i wrote a script to capture the document.  Actually, i wrote two scripts.  The first script uses xdotool and scrot (a fast and script-friendly screenshot tool) to click through the document and capture all the image data.  It helps to have a large monitor, otherwise, one can zoom in and take multiple shots per page.  

#!/bin/bash
# click through drm'd digibooks in brytewave reader and copy them as screenshots

clickwait=1       
pagewait=20    # seconds to wait for next page to load
startpage=1     # useful in case of crash (it happens)
endpage=400   # the last reader page 
dumppath="/home/personface/pileofshittybooks"
docname="governmentbook"

xdotool search --screen 0 --name "BryteWave" windowfocus; 
page=$startpage
while [ $page -le $endpage ]; do
    scrot "$dumppath/$docname-$page.jpg"

    # use a screenshot to find button coordinates    
    sleep $clickwait; xdotool mousemove 1907 630
    sleep 0.1; xdotool click 1
    sleep $pagewait
    
    page=$[page+1]
done

The second script cuts the two pages out of each screenshot and compiles a minimal pdf from the images.

#!/bin/bash
# disassemble screenshots made of bryteclicker

docprefix="governmentbook"
pdfname="Fascist_Propaganda_by_CKSucker_10ed"

inext="jpg"
outext="jpg"
outqual=50 
endonduplicate=1

page=1
lastsize=0
for infile in $(ls -1tr $docprefix*.$inext); do
    # check for consecutive duplicates since screengrabber cannot verify page loads
    # if flag is set, assume duplicates indicate screengrabber is stalled on last page of document
    # useful when screengrabber doesn't know exact document size and is set with excess pagecount
    thissize=$(ls -l $infile | cut -d " " -f 5)
    if [ $lastsize == $thissize ]; then
        echo "$infile may be a duplicate of the previous file!"
        if [ $endonduplicate == 1 ]; then
            echo "i'm going to assume this is the end of the document"
            break;
        fi
    fi
    lastsize=$thissize

    # crop pages from screenshot
    # use GIMP to get coordinates
    convert -crop 670x1005+282+122 -quality $outqual $infile "output-$(printf "%04d" $page)-A.$outext"
    convert -crop 670x1005+968+122 -quality $outqual $infile "output-$(printf "%04d" $page)-B.$outext"
    page=$[$page+1]
done

convert output*.$outext $pdfname.pdf

Like i mentioned, one can get better image quality by making each page span more than one screenshot, but this complicates post-processing a tad.  The quality settings in the second script can also be tweaked for better output.  My method was a tradeoff between readability and file size reduction.

Audacious: copy current selection to next playlist

Audacious has two different layout options.  One is the familiar classic winamp/xmms style, whereas the other is a more featureful gtk UI.  Under Audacious 3.2.3, the gtk UI exhibits severe memory leaks.  Although newer versions of Audacious have had this problem fixed, they do not run on my current system without other more spectacular usability failures.  Until such a time as i upgrade, I'm stuck using the classic UI under 3.2.3.  I'm okay with that.  I guess i still carry a torch for good old xmms anyway.

Then there's the problem at hand: manipulating playlists in the classic UI is cumbersome as hell.  I only have two playlists: one which contains all files, and one in the background which contains favorites.  I want to easily copy a song i like from one to the other without a bunch of menus or keymashing.

I wrote a simple script based on xdotool:

#!/bin/bash
# this script externally issues commands required to copy tracks 
# from the main playlist to the secondary "favorites" playlist
# without invoking the GTK UI with bad memory leaks
# using windowactivate instead of windowfocus causes cross-$DISPLAY bugginess

xdotool search --screen 1 --name "Audacious Playlist" windowfocus; 
sleep 0.1; xdotool key Ctrl+c
sleep 0.1; xdotool key Tab
sleep 0.1; xdotool key Ctrl+v
sleep 0.1; xdotool key Shift+Tab

I then stuck it in a panel launcher on the same X display.  Now all i need to do is select the songs and click the panel button. I suppose one could use a global keybinding instead.

Make an application stay on top by default

Certain applications such as a calculator have typical use patterns which indicate window behavior settings contrary to the norm.  A calculator application like gcalctool is often used alongside work being performed in other windows. In these cases, i prefer the window mode to be set such that it stays on top of all other windows.  This way i can go back and forth between the calculator and other work without the results of my calculations disappearing under a stack of other windows.  This is only more of a hassle since i have the window manager set to auto-focus and auto-raise windows.  Since the application may be opened and closed often, efforts to manually set window modes are usually wasted.

wmctrl can be used to set the mode flags of a window (among a bunch of other things).
For gcalctool:

#!/bin/bash

gcalctool &
sleep 1
wmctrl -r "Calculator" -b add,above

For xzoom magnifier:

#!/bin/bash

xzoom &
sleep 1
wmctrl -r "xzoom x2" -b add,above,sticky

this time, including the sticky flag

Fix black missing window decorations in xfwm4

This is one of many problems i seem to have a problem discussing with anyone due to the growing rarity of Zaphod X configurations. In my configuration, i have three screens on different xorg servers (no xnest).  Using Mint14 and XFCE, when i log into my desktop environment, there are no window decorations (title bars, buttons, frames) drawn for the windows on displays :0.1 or :0.2. Only the first display (:0) has any window decorations; everything else is black.

Left: Everything is fine.  Right: Windeco's are missing

In my case, going to the Window Manager settings dialog and cycling between themes was sufficient to fix the issue.  If GTK settings are altered, the fix will need to be re-applied.  In order to simplify and automate this workaround, I  slapped a script together that uses xfconf-query to cycle to a different xfwm4 theme and back:

#!/bin/bash
# resets the xfwm4 window decoration theme
# this is a workaround to help enforce drawing on secondary $DISPLAYs

xfconf-query -c xfwm4 -p /general/theme -s Tgc
sleep 0.5
xfconf-query -c xfwm4 -p /general/theme -s Next

Stick this script wherever is convenient for your session startup needs.  I use it in a session template script, but there are more conventional places to put it.