Commit Graph
25 Commits
Author SHA1 Message Date
Klaus Postandkannappanr 1c90a6bd49 S3 Select: Convert CSV data to JSON (#8464) 2019-11-09 09:10:35 -08:00
Klaus PostandNitish Tiwari 38e6d911ea S3 Select: Detect full object (#8456)
Check if select is `SELECT s.* from S3Object s` and forward it to All

Fixes #8371 and makes this case run significantly faster.
2019-10-30 13:46:55 +05:30
Klaus PostandNitish Tiwari 51456e6adc Select: Support Square Bracket Lists (#8457)
Allows for S3 compatible `SELECT * from s3object s WHERE id IN [3,2]`

Fixes #8422
2019-10-30 11:34:40 +05:30
Klaus PostandHarshavardhana 002ac82631 S3 Select: Add parser support for lists. (#8329) 2019-10-06 07:52:45 -07:00
Klaus PostandHarshavardhana c1a17c2561 S3 Select: Aggregate AVG/SUM as float (#8326)
Force sum/average to be calculated as a float.

As noted in #8221

> run SELECT AVG(CAST (Score as int)) FROM S3Object on

```
Name,Score
alice,80
bob,81
```

> AWS S3 gives 80.5 and MinIO gives 80.

This also makes overflows much more unlikely.
2019-09-27 16:12:03 -07:00
Klaus PostandHarshavardhana 1c5b05c130 S3 select: Fix output conversion on select * (#8303)
Fixes #8268
2019-09-27 12:33:14 -07:00
Klaus PostandHarshavardhana dac1cf5a9a S3 Select: Parsing tweaks (#8261)
* Don't output empty lines.
* Trim whitespace from byte to int/float/bool conversions.
2019-09-17 17:21:23 -07:00
Klaus Postandkannappanr c9b8bd8de2 S3 Select: optimize output (#8238)
Queue output items and reuse them.
Remove the unneeded type system in sql and just use the Go type system.

In best case this is more than an order of magnitude speedup:

```
BenchmarkSelectAll_1M-12    	       1	1841049400 ns/op	274299728 B/op	 4198522 allocs/op
BenchmarkSelectAll_1M-12    	      14	  84833400 ns/op	169228346 B/op	 3146541 allocs/op
```
2019-09-17 05:56:27 +05:30
Yao ZongyouandHarshavardhana ec9bfd3aef speed up the performance of s3select on csv (#7945) 2019-08-31 00:07:40 -07:00
Yao ZongyouandHarshavardhana 60831e3299 aggregation functions' argument may already has been cast to numeric (#7876) 2019-07-05 10:38:38 -07:00
Yao ZongyouandHarshavardhana 037319066f fix unicode support related bugs in s3select (#7877) 2019-07-05 09:43:10 -07:00
Ryan Tamandkannappanr bd56f80250 Fix ignored alias for aggregate result in S3 Select (#7849)
The SQL parser as it stands right now ignores alias for aggregate
result, e.g. `SELECT COUNT(*) AS thing FROM s3object` doesn't actually
return record like `{"thing": 42}`, it returns a record like `{"_1": 42}`.
Column alias for aggregate result is supported in AWS's S3 Select, so
this commit fixes that by respecting the `expr.As` in the expression.

Also improve test for S3 select

On top of testing a simple `SELECT` query, we want to test a few more
"advanced" queries (e.g. aggregation).

Convert existing tests into table driven tests[1], and add the new test
cases with "advanced" queries into them.

[1] - https://github.com/golang/go/wiki/TableDrivenTests
2019-07-03 16:34:54 -07:00
Yao Zongyouandkannappanr 55092bede1 add timestamp compare support (#7832) 2019-06-25 11:05:37 -07:00
Yao Zongyouandkannappanr 90a3b830f4 fix typo and the string representation of the time.Time value (#7831) 2019-06-25 09:54:14 -07:00
Yao Zongyouandkannappanr 23b9df0694 Fix s3select TRIM function's nil pointer dereference bug (#7817) 2019-06-24 16:59:33 -07:00
kannappanrandGitHub 5ecac91a55 Replace Minio refs in docs with MinIO and links (#7494) 2019-04-09 11:39:42 -07:00
Kirill Motkovandkannappanr 3d29ab4059 Rewrite if-else chains to switch statements (#7382) 2019-03-18 07:46:20 -07:00
Aditya ManthramurthyandHarshavardhana e463386921 Add JSON Path expression evaluation support (#7315)
- Includes support for FROM clause JSON path
2019-03-09 08:13:37 -08:00
Aditya Manthramurthyandkannappanr 8a405cab2f COUNT() function in select should return an int (#7243) 2019-02-13 16:32:59 -08:00
HarshavardhanaandNitish Tiwari df35d7db9d Introduce staticcheck for stricter builds (#7035) 2019-02-13 18:29:36 +05:30
Aditya Manthramurthyandkannappanr ee5b3622a5 Evaluate where clause in aggregation queries (#7235) 2019-02-12 13:54:26 -08:00
Aditya Manthramurthyandkannappanr f04f8bbc78 Add support for Timestamp data type in SQL Select (#7185)
This change adds support for casting strings to Timestamp via CAST:
`CAST('2010T' AS TIMESTAMP)`

It also implements the following date-time functions:
  - UTCNOW()
  - DATE_ADD()
  - DATE_DIFF()
  - EXTRACT()

For values passed to these functions, date-types are automatically
inferred.
2019-02-04 20:54:45 -08:00
Aditya ManthramurthyandHarshavardhana 2786055df4 Add new SQL parser to support S3 Select syntax (#7102)
- New parser written from scratch, allows easier and complete parsing
  of the full S3 Select SQL syntax. Parser definition is directly
  provided by the AST defined for the SQL grammar.

- Bring support to parse and interpret SQL involving JSON path
  expressions; evaluation of JSON path expressions will be
  subsequently added.

- Bring automatic type inference and conversion for untyped
  values (e.g. CSV data).
2019-01-28 17:59:48 -08:00
Bala FAandNitish Tiwari e23a42305c Rebase minio/parquet-go and fix null handling. (#7067) 2019-01-16 21:52:04 +05:30
Bala FAandkannappanr b0deea27df Refactor s3select to support parquet. (#7023)
Also handle pretty formatted JSON documents.
2019-01-08 16:53:04 -08:00