Questions tagged [awk]
AWK is an interpreted programming language designed for text processing and typically used as a data extraction and reporting tool.
56
questions
3
votes
2
answers
130
views
Processing a very long single line of comma-separated (?) floating point numbers
I have a sample svg file from the graphics program Inkscape. My objective is to collect every third pair from a very long line of floating point numbers. That seems simple enough. The larger objective ...
3
votes
1
answer
36
views
Split Text to Columns and move all data to the next column
Objective
I'd like to confirm if there is a more efficient approach for the working script below, which splits column data in my tsv files.
I suspect this can be achieved with awk or maybe sed but my ...
- 131
4
votes
1
answer
124
views
Awk program for extracting unique values from a k1=v1,k2=v2,... list
A single string consisting of comma-separated pairs in the format <key>=<value>, like
...
- 143
1
vote
1
answer
40
views
Extract sections of a file into separate files
I have a file of the form -
>SDF123.1 blah blah
ATCTCTGGAAACTCGGTGAAAGAGAGTAT
AGTGATGAGGATGAGTGAG...
>SBF123.1 blah blah
ATCTCTGGAAACTCGGTGAAAGAGAGTAT
AGTGATGAGGATGAGTGAG....
And I want to ...
- 135
0
votes
0
answers
49
views
UNIX calendar(1) in awk
This is a simple implementation of the calendar(1) utility included in some UNIX systems (all BSDs have it, GNU has not).
Here's the manual:
...
- 864
1
vote
0
answers
42
views
detection the selfish node Using awk File [closed]
I am working on detection the selfish node based on this equation Using awk File : (Node Communication Ratio) NCR= ((Get Route Request-(Get Route Request-Reply Route Request))/Get Route Request)*100
...
- 19
3
votes
0
answers
73
views
The Blacklist v2
This script gathers domains listed from sources in this JSON file and compiles them into one text file. Text files that have a similar key under the whitelists ...
- 3,087
3
votes
1
answer
139
views
Parse log file and send the result to an API
The task is to get the most recent login time of the day for each user. Then send such a result to an API.
The log file (file.log) looks like the following (only ...
- 5,522
2
votes
0
answers
46
views
bash / awk workflow operating on multi-column files [closed]
I am working on the improvement of the bash-awk workflow designed for the analysis of data in multiple-colums format. The script uses AWK code (which extract numbers from selected columns of the input....
- 149
2
votes
1
answer
73
views
Printing of status message during execution of the bash script
As a part of my bash routine I am printing some message in terminal regarding status of the workflow. The message in splited into two parts (part 1: begining of the task, part 2: status of its ...
- 149
0
votes
1
answer
108
views
Compare script using awk and bash
I try to create compare script using awk and another command and running successful.
But I think the script I create it's to long.
Is there anyone can shorten my script below ?
After plan A ...
- 11
2
votes
0
answers
51
views
Faster algorithm/language for substring sliding window replacement
I had a sample file with a long string >50M like this.
...
- 21
2
votes
1
answer
314
views
Bash script wallpaper randomizer
I thought it would be fun to write a simple and small bash script which randomly chooses a wallpaper from a given directory of a set of wallpaper images. The code is as follows.
...
user228607
1
vote
1
answer
75
views
Parse and clean large log files
I have the code below, which works successfully, and is used to parse, clean log files (very large in size) and output into smaller sized files. Output filename is the first 2 characters of each line. ...
- 181
0
votes
1
answer
28
views
Split large log file like a dictionary [closed]
I have a large log file in the following format. Do note that this is already sorted alphabetically. I wish to split it into smaller pieces, and with file names reflecting the content of the file (...
- 181
2
votes
2
answers
98
views
Extracting emails from log files
I have huge number of text files, each several hundred MB in size. Unfortunately, they are not all fully standardized in any one format. Plus there is a lot of legacy in here, and a lot of junk and ...
- 181
3
votes
1
answer
544
views
An Awk program to find all primes till N
I developed Awk code to print all primes till N where N is given as command line argument. I think I took pretty much all the optimizations into account. Could someone help if it could be optimized ...
- 131
3
votes
0
answers
38
views
Merging multiple awk commands while retaining functionality
Turned in a project a few days ago, and I needed a way to filter and format values off a file. I decided to go for a bash script since it seemed like a good option, and I came up with the following:
<...
- 31
3
votes
1
answer
2k
views
Extract multiple attributes values in XML tags
I have a XML file (test.xml) that can be sum up as follow (I filtered it so it will be more readable):
...
- 287
2
votes
0
answers
153
views
Bash wrapper for connmanctl with autocompletion
So the idea came from the fact that
connmanctl only completes filenames/directories when used non-interactively,
whereas when used interactively (executing ...
- 547
6
votes
3
answers
544
views
Count unique IP address in a date range from Apache log
I use the following command to get specific entries between certain dates from Apache log files. Then I filter out my own address and some bots, then I print out the IP, sort it, get the uniques, ...
- 265
4
votes
1
answer
81
views
Removing duplicate field entries from sorted csv data
Given the following input (cat i.txt),
I want to remove duplicate field entries in each of the first three columns and none of the others.
...
0
votes
2
answers
96
views
Select a specific line after matching a pattern
I just recently started using AWK and I'm still learning about it. I have solved the problem I'm about to show but I feel it's not the best solution and I'm trying to find a solution to fit within an ...
- 3
4
votes
1
answer
91
views
Select second field in text table with shell script
I am writing a bash script to mount openshift service accounts into kubernetes objects. The right secret to use is highlighted in this text:
$ oc describe sa sa-...
- 308
3
votes
2
answers
117
views
Code to get the IP address and format it to run in nmap
For scanning my subnet using my IP address, I need to run the following command
nmap -sn 192.168.100.*
I want to extend this functionality such that I don't have ...
- 243
5
votes
0
answers
53
views
Summary data using CSV file
Is there the possibility to improve the code below?
I am sure the result I got can be achieved with shorter code.
Code
...
- 183
7
votes
1
answer
2k
views
Updating my package version using only a Makefile
So I have written a Makefile that has a release function that can be run like: make release bump=patch and I want it to be fully ...
- 243
7
votes
3
answers
649
views
Print code-fenced sections of a Markdown document
Original code and demo at this gist.
Given a Markdown document like
...
- 1,169
3
votes
1
answer
56
views
Counts and average code
Using the columns 4 and 2 will create a report like the output file showed below. My code works fine but I believe it can be done shorter.
I have a doubt in the part of the split:
...
- 183
3
votes
2
answers
319
views
Bash script for managing hashtag notes
I have written a simple script for managing hashtag notes.
I am not familiar with bash scripting so I would really appreciate it if someone could give me some pointers about it.
...
- 33
4
votes
1
answer
56
views
Remove every line that include and follows the second occurrence of a given pattern
From a file, I am trying to remove every line that include and follows the second occurrence of a given pattern in Bash (Mac OSX). Note, that a file may have only one match of the given pattern (and ...
- 617
4
votes
1
answer
117
views
Try every option for a command and make anagrams out of the successful ones
What I have: I am trying to find what words you can make out of Linux command options like ls -cat or ps -elf. To do that I am ...
- 143
2
votes
1
answer
177
views
Extract thoughput value from JMeter log file
I use JMeter for load testing and want to report error on CI pipeline when application though put is less than 1/sec. JMeter logs summary of results as -
...
- 165
4
votes
1
answer
117
views
ECG Bash selection tool
I made the following bash script for extracting a group of ECG signals from ECG files. I would like to know if there is any mistakes and/or weaknesses. I have experienced difficulties in integrating ...
5
votes
1
answer
3k
views
Python library for awk-like file manipulation
I recently published a library for advanced awk-like file manipulation in Python 3. The code can be found here and here is the documentation. It is also available for download from pip (...
- 103
3
votes
1
answer
519
views
Parsing GTF file using command-line
I am extracting exons details from a GTF file using command line in Unix like cut, awk, grep or sed.
input file.gtf:
...
- 131
4
votes
2
answers
448
views
Generate SQL UPDATE from Excel CSV file
I must write updates in a database from a flat file (CSV). I want to do that in the shell, with tools such as AWK.
...
- 169
4
votes
1
answer
259
views
Find and report all occurrences of duplicate lines in a text file
I would like you to take a look at this simple script I wrote that's supposed to emulate the Unix sort [FILE] | uniq -cd command. What's different about my script ...
- 297
0
votes
1
answer
46
views
Loading a file in vim via piped output of ag
I feel as if my scenario is fairly simple but that I might be overcomplicating it.
I am searching through files looking for a special condition, that is where the method ...
- 439
2
votes
1
answer
703
views
Finding hidden field values in a web page before submitting a login form
In order to submit a login form using wget, I need to find the values for the hidden field that were added to the form recently. Grepping for the names gives the ...
- 341
2
votes
1
answer
64
views
Listing Docker Images without registry URI
I have a private container registry behind a firewall that will need some Google registry images. To make sure things are updated sanely, I'm writing a mirror script in Bash. I have one line that ...
- 453
3
votes
2
answers
257
views
Advanced CSV-to-SQLite converter
I just wrote an advanced CSV parser that translates CSV to SQL(SQLite Compatible) statements, which are then inserted into the database. It translates variable names in the CSV to values defined in ...
- 875
1
vote
1
answer
39
views
Print first element of the last line of files when the second element does not match a pattern
I am relatively new to Bash, regular expression and stuff.
In a directory, I need to read the last line of all the files that contain the word "fitness", and list (on one line and separated with ...
- 617
6
votes
3
answers
335
views
Shell script to count chess game outcomes
I came across this blog post by Adam Drake from around a year ago which is now making the rounds again.
I made some improvements to his code, but wish to see if there are additional tweaks that could ...
- 423
7
votes
1
answer
78
views
5
votes
2
answers
3k
views
Checking percentage of free memory using top and awk
The following Awk code was created in order to verify if free memory from the top command is less than 20% of the total.
I've noticed that I actually do not need the ...
- 53
3
votes
1
answer
2k
views
Analyse an Nginx access log with awk and produce a report
I'm looking for your opinions on anything that I am doing wrong in the application below, such as best practices, glaringly horrible errors or even just your own personal opinion.
Task: Grab an Nginx ...
- 221
10
votes
3
answers
26k
views
Extracting data from text file in bash using awk, grep, head and tail
I've been writing bash script on and off, with pretty good results in terms of getting the job done. However, I'm worried that my script might be very ugly, as I am a beginner. I'm looking for advice ...
- 103
4
votes
2
answers
117
views
Filter shell script to find lines that contain all specified patterns
I wrote a script that does the following:
Run another script on the system
Filter the output to find lines that contain ALL of the given patterns
Pipe the output to a second script on the system
I ...
- 918
4
votes
1
answer
76
views
Extracting performance statistics from FTP session transcript
I've borrowed and written the following code to output the disconnect time. All works well but I'm curious as to how I could tighten/ shorten the code. If anyone feels like having some fun then I'd ...
- 41