Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [awk]

AWK is an interpreted programming language designed for text processing and typically used as a data extraction and reporting tool.

Filter by
Sorted by
Tagged with
4 votes
1 answer
182 views

awk script to pivot long CSV into wide CSV

I've made an awk script that does the same operation and has the same parameters as Python's pandas.DataFrame.pivot, for 2D tables. It reshapes a table from "...
minseong's user avatar
  • 749
4 votes
2 answers
135 views

Processing a very long single line of comma-separated (?) floating point numbers

I have a sample svg file from the graphics program Inkscape. My objective is to collect every third pair from a very long line of floating point numbers. That seems simple enough. The larger objective ...
Bryan-StackExchange's user avatar
3 votes
1 answer
42 views

Split Text to Columns and move all data to the next column

Objective I'd like to confirm if there is a more efficient approach for the working script below, which splits column data in my tsv files. I suspect this can be achieved with awk or maybe sed but my ...
El_Birdo's user avatar
  • 131
4 votes
1 answer
138 views

Awk program for extracting unique values from a k1=v1,k2=v2,... list

A single string consisting of comma-separated pairs in the format <key>=<value>, like ...
Amaterasu's user avatar
  • 205
1 vote
1 answer
40 views

Extract sections of a file into separate files

I have a file of the form - >SDF123.1 blah blah ATCTCTGGAAACTCGGTGAAAGAGAGTAT AGTGATGAGGATGAGTGAG... >SBF123.1 blah blah ATCTCTGGAAACTCGGTGAAAGAGAGTAT AGTGATGAGGATGAGTGAG.... And I want to ...
user1995's user avatar
  • 135
1 vote
0 answers
42 views

detection the selfish node Using awk File [closed]

I am working on detection the selfish node based on this equation Using awk File : (Node Communication Ratio) NCR= ((Get Route Request-(Get Route Request-Reply Route Request))/Get Route Request)*100 ...
sally's user avatar
  • 19
3 votes
0 answers
77 views

The Blacklist v2

This script gathers domains listed from sources in this JSON file and compiles them into one text file. Text files that have a similar key under the whitelists ...
T145's user avatar
  • 3,099
3 votes
1 answer
143 views

Parse log file and send the result to an API

The task is to get the most recent login time of the day for each user. Then send such a result to an API. The log file (file.log) looks like the following (only ...
Marc's user avatar
  • 5,604
2 votes
0 answers
47 views

bash / awk workflow operating on multi-column files [closed]

I am working on the improvement of the bash-awk workflow designed for the analysis of data in multiple-colums format. The script uses AWK code (which extract numbers from selected columns of the input....
Hot JAMS's user avatar
  • 149
2 votes
1 answer
81 views

Printing of status message during execution of the bash script

As a part of my bash routine I am printing some message in terminal regarding status of the workflow. The message in splited into two parts (part 1: begining of the task, part 2: status of its ...
Hot JAMS's user avatar
  • 149
0 votes
1 answer
113 views

Compare script using awk and bash

I try to create compare script using awk and another command and running successful. But I think the script I create it's to long. Is there anyone can shorten my script below ? After plan A ...
Ario's user avatar
  • 11
2 votes
0 answers
51 views

Faster algorithm/language for substring sliding window replacement

I had a sample file with a long string >50M like this. ...
hey0god's user avatar
  • 21
2 votes
1 answer
359 views

Bash script wallpaper randomizer

I thought it would be fun to write a simple and small bash script which randomly chooses a wallpaper from a given directory of a set of wallpaper images. The code is as follows. ...
user avatar
1 vote
1 answer
78 views

Parse and clean large log files

I have the code below, which works successfully, and is used to parse, clean log files (very large in size) and output into smaller sized files. Output filename is the first 2 characters of each line. ...
rogerwhite's user avatar
0 votes
1 answer
28 views

Split large log file like a dictionary [closed]

I have a large log file in the following format. Do note that this is already sorted alphabetically. I wish to split it into smaller pieces, and with file names reflecting the content of the file (...
rogerwhite's user avatar
2 votes
2 answers
101 views

Extracting emails from log files

I have huge number of text files, each several hundred MB in size. Unfortunately, they are not all fully standardized in any one format. Plus there is a lot of legacy in here, and a lot of junk and ...
rogerwhite's user avatar
3 votes
1 answer
576 views

An Awk program to find all primes till N

I developed Awk code to print all primes till N where N is given as command line argument. I think I took pretty much all the optimizations into account. Could someone help if it could be optimized ...
Ketan's user avatar
  • 131
3 votes
0 answers
39 views

Merging multiple awk commands while retaining functionality

Turned in a project a few days ago, and I needed a way to filter and format values off a file. I decided to go for a bash script since it seemed like a good option, and I came up with the following: <...
rubbun's user avatar
  • 31
3 votes
1 answer
2k views

Extract multiple attributes values in XML tags

I have a XML file (test.xml) that can be sum up as follow (I filtered it so it will be more readable): ...
Nuageux's user avatar
  • 287
2 votes
0 answers
167 views

Bash wrapper for connmanctl with autocompletion

So the idea came from the fact that connmanctl only completes filenames/directories when used non-interactively, whereas when used interactively (executing ...
Enlico's user avatar
  • 547
6 votes
3 answers
598 views

Count unique IP address in a date range from Apache log

I use the following command to get specific entries between certain dates from Apache log files. Then I filter out my own address and some bots, then I print out the IP, sort it, get the uniques, ...
Tschallacka's user avatar
4 votes
1 answer
82 views

Removing duplicate field entries from sorted csv data

Given the following input (cat i.txt), I want to remove duplicate field entries in each of the first three columns and none of the others. ...
Thure Dührsen's user avatar
0 votes
2 answers
104 views

Select a specific line after matching a pattern

I just recently started using AWK and I'm still learning about it. I have solved the problem I'm about to show but I feel it's not the best solution and I'm trying to find a solution to fit within an ...
zuNEC's user avatar
  • 3
4 votes
1 answer
92 views

Select second field in text table with shell script

I am writing a bash script to mount openshift service accounts into kubernetes objects. The right secret to use is highlighted in this text: $ oc describe sa sa-...
simbo1905's user avatar
  • 308
3 votes
2 answers
123 views

Code to get the IP address and format it to run in nmap

For scanning my subnet using my IP address, I need to run the following command nmap -sn 192.168.100.* I want to extend this functionality such that I don't have ...
user3079474's user avatar
5 votes
0 answers
53 views

Summary data using CSV file

Is there the possibility to improve the code below? I am sure the result I got can be achieved with shorter code. Code ...
OXXO's user avatar
  • 183
7 votes
1 answer
2k views

Updating my package version using only a Makefile

So I have written a Makefile that has a release function that can be run like: make release bump=patch and I want it to be fully ...
Rambatino's user avatar
  • 243
7 votes
3 answers
660 views

Print code-fenced sections of a Markdown document

Original code and demo at this gist. Given a Markdown document like ...
wchargin's user avatar
  • 1,169
3 votes
1 answer
57 views

Counts and average code

Using the columns 4 and 2 will create a report like the output file showed below. My code works fine but I believe it can be done shorter. I have a doubt in the part of the split: ...
OXXO's user avatar
  • 183
3 votes
2 answers
353 views

Bash script for managing hashtag notes

I have written a simple script for managing hashtag notes. I am not familiar with bash scripting so I would really appreciate it if someone could give me some pointers about it. ...
Kanzey's user avatar
  • 33
4 votes
1 answer
56 views

Remove every line that include and follows the second occurrence of a given pattern

From a file, I am trying to remove every line that include and follows the second occurrence of a given pattern in Bash (Mac OSX). Note, that a file may have only one match of the given pattern (and ...
Remi.b's user avatar
  • 617
4 votes
1 answer
123 views

Try every option for a command and make anagrams out of the successful ones

What I have: I am trying to find what words you can make out of Linux command options like ls -cat or ps -elf. To do that I am ...
prolificslacker's user avatar
2 votes
1 answer
178 views

Extract thoughput value from JMeter log file

I use JMeter for load testing and want to report error on CI pipeline when application though put is less than 1/sec. JMeter logs summary of results as - ...
Tarun's user avatar
  • 175
4 votes
1 answer
124 views

ECG Bash selection tool

I made the following bash script for extracting a group of ECG signals from ECG files. I would like to know if there is any mistakes and/or weaknesses. I have experienced difficulties in integrating ...
Léo Léopold Hertz 준영's user avatar
5 votes
1 answer
3k views

Python library for awk-like file manipulation

I recently published a library for advanced awk-like file manipulation in Python 3. The code can be found here and here is the documentation. It is also available for download from pip (...
Simone Bronzini's user avatar
3 votes
1 answer
562 views

Parsing GTF file using command-line

I am extracting exons details from a GTF file using command line in Unix like cut, awk, grep or sed. input file.gtf: ...
Zingo's user avatar
  • 131
4 votes
2 answers
460 views

Generate SQL UPDATE from Excel CSV file

I must write updates in a database from a flat file (CSV). I want to do that in the shell, with tools such as AWK. ...
user3341592's user avatar
4 votes
1 answer
269 views

Find and report all occurrences of duplicate lines in a text file

I would like you to take a look at this simple script I wrote that's supposed to emulate the Unix sort [FILE] | uniq -cd command. What's different about my script ...
misha's user avatar
  • 297
0 votes
1 answer
48 views

Loading a file in vim via piped output of ag

I feel as if my scenario is fairly simple but that I might be overcomplicating it. I am searching through files looking for a special condition, that is where the method ...
Anthony's user avatar
  • 439
2 votes
1 answer
704 views

Finding hidden field values in a web page before submitting a login form

In order to submit a login form using wget, I need to find the values for the hidden field that were added to the form recently. Grepping for the names gives the ...
Xavier Nodet's user avatar
2 votes
1 answer
66 views

Listing Docker Images without registry URI

I have a private container registry behind a firewall that will need some Google registry images. To make sure things are updated sanely, I'm writing a mirror script in Bash. I have one line that ...
Sienna's user avatar
  • 453
3 votes
2 answers
263 views

Advanced CSV-to-SQLite converter

I just wrote an advanced CSV parser that translates CSV to SQL(SQLite Compatible) statements, which are then inserted into the database. It translates variable names in the CSV to values defined in ...
HSchmale's user avatar
  • 895
1 vote
1 answer
39 views

Print first element of the last line of files when the second element does not match a pattern

I am relatively new to Bash, regular expression and stuff. In a directory, I need to read the last line of all the files that contain the word "fitness", and list (on one line and separated with ...
Remi.b's user avatar
  • 617
6 votes
3 answers
351 views

Shell script to count chess game outcomes

I came across this blog post by Adam Drake from around a year ago which is now making the rounds again. I made some improvements to his code, but wish to see if there are additional tweaks that could ...
tripleee's user avatar
  • 491
7 votes
1 answer
83 views

Calculating percentages in arbitrary number of columns

Given this sample input: ...
janos's user avatar
  • 111k
5 votes
2 answers
3k views

Checking percentage of free memory using top and awk

The following Awk code was created in order to verify if free memory from the top command is less than 20% of the total. I've noticed that I actually do not need the ...
maihabunash's user avatar
3 votes
1 answer
2k views

Analyse an Nginx access log with awk and produce a report

I'm looking for your opinions on anything that I am doing wrong in the application below, such as best practices, glaringly horrible errors or even just your own personal opinion. Task: Grab an Nginx ...
Ne Ma's user avatar
  • 221
10 votes
3 answers
26k views

Extracting data from text file in bash using awk, grep, head and tail

I've been writing bash script on and off, with pretty good results in terms of getting the job done. However, I'm worried that my script might be very ugly, as I am a beginner. I'm looking for advice ...
Pierre's user avatar
  • 103
4 votes
2 answers
117 views

Filter shell script to find lines that contain all specified patterns

I wrote a script that does the following: Run another script on the system Filter the output to find lines that contain ALL of the given patterns Pipe the output to a second script on the system I ...
durron597's user avatar
  • 918
4 votes
1 answer
76 views

Extracting performance statistics from FTP session transcript

I've borrowed and written the following code to output the disconnect time. All works well but I'm curious as to how I could tighten/ shorten the code. If anyone feels like having some fun then I'd ...
ten1267's user avatar
  • 41