bash - awk multiple passes on each line -

I have to process some large files and run multiple tests on each line. I am currently using awk to run the test individually and I am using the "while-read-line-do" loop to pass each line in more than a dozen awk commands, To validate their content and to test the errors, the lines passing all the tests have been added to the .VALID file.

The problem is that I am currently in the process that the process is very slow around the web and after reading many other posts on the stack overflow, what I collect is that the main culprit "when the read-line Is a "loop" which does not lend memory to files (about 100K lines each) in memory.

I was hoping someone could help me understand a better way to implement things so that I could get an awk-like performance. Here's my simplified version of code:

  line while reading || [[-n "$ line"]]; Echo $ line Awk -F \; '{If (($ 3! = "P") & amp; amp; ($ 3! = "0")) {Print $ 0 "; ERROR;" & Gt; & Gt; "INPUT_FILE.ERRORS"}; Other print $ 0 & gt; & Gt; "INPUT_FILE.OK"; } 'Echo $ line | Awk -F \; '{If (($ 7> 10) || ($ 7 & 3;)} {Print $ 0 "; ERROR;" & Gt; & Gt; "INPUT_FILE.ERRORS"}; Other print $ 0 & gt; & Gt; "INPUT_FILE.OK"; } 'Echo $ line | Awk -F \; '{If (($ 36 & lt; 0)} ($ 36>, 1000)) {Print $ 0 "; ERROR;" & Gt; & Gt; "INPUT_FILE.ERRORS"}; Other print $ 0 & gt; & Gt; "INPUT_FILE.OK"; } 'Done & lt; INPUT_FILE.txt  

Ideally I'm trying to come up with a solution that gives me multiple passes per line using an awk-based loop.

Thanks in advance.

Absolutely not pass lines on awk after one; Awk processes file lines by line on your behalf. The code in your reply can be reduced to:

  awk -F \; ($ 3! = "P" and $ 3! = "0") || ($ 7 & 10; $ 10 || $ 7 & gt; 3) || ($ 36 & lt; 0 = $ 36> 1000) {Print $ 0 "; Error;" & Gt; & Gt; "INPUT_FILE.ERRORS"; Next} {Print & gt; & Gt; "INPUT_FILE.OK"} 'INPUT_FILE.txt  

I suspect it will be very fast.

The structure of an awk program is condition {action} , so it's hardly the case that you if / else Instead, you can use the if branch in next , which means that the awk will move to the next line instead of the second block .

The output will be slightly different because the lines failing more than one test will not be duplicated in the error log. I think it was okay, because the output was the same for each of your checks.

For further improvement in performance, you can consider arranging trials in the order of equality, because this will mean that the possibility of having a short circuit is high.

Note that awk in Shell, & gt; and & gt; & Gt; has different meanings. & gt; means that awk creates a new file for the first time and adds it on continuous typing, so you may want to use it. If the file does not already exist, then it really does not matter.

As noted in the observations, it seems that $ 7 & lt; 10 || $ 7 & gt; There is a logical error with; 3 because it is always true, maybe you have & gt; and & lt; Found?

If you want to write separate output for each error, then you can change the structure slightly to do something like this:

  awk -F \; '$ F = 0} $ 3! = "P" & amp; $ 3! = "0" {Print $ 0 "; Error;" & Gt; & Gt; "INPUT_FILE.ERRORS"; F = 1} $ 7 & lt; 3 || $ 7 & gt; 10 {Print $ 0 "; Error;" & Gt; & Gt; "INPUT_FILE.ERRORS"; F = 1} $ 36 & lt; 0 || $ 36 & gt; 1000 {Print $ 0 "; ERROR;" & Gt; & Gt; "INPUT_FILE.ERRORS"; F = 1}! F {Print & gt; & Gt; "INPUT_FILE.OK"} 'INPUT_FILE.txt  

Each test is made separately and if any test is correct then f is set to true. If the f is still incorrect after all the tests on the line, then it is printed in the correct file. I also changed my second exam so that it is not always right.