前提條件

注意事項

使用限制

OSS外部表不支持cluster屬性。

操作入口

創建OSS外部表語法

創建OSS外部表的場景、相應語法格式及示例如下。詳細語法參數及屬性列表信息，請參見參考：語法參數說明、參考：with serdeproperties屬性列表和參考：tblproperties屬性列表。

CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_extable_name> 
(
<col_name> <data_type>,
...
)
[partitioned BY (<col_name> <data_type>, ...)] 
stored BY '<StorageHandler>'  
WITH serdeproperties (
 ['<property_name>'='<property_value>',...]
) 
location '<oss_location>';

CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_extable_name>
(
<col_name> <data_type>,
...
)
[partitioned BY (<col_name> <data_type>, ...)]
[row format serde '<serde_class>'
  [WITH serdeproperties (
    ['<property_name>'='<property_value>',...])
  ]
]
stored AS <file_format> 
location '<oss_location>' 
[USING '<resource_name>']
[tblproperties ('<tbproperty_name>'='<tbproperty_value>',...)];

CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_extable_name> 
(
<col_name> <date_type>,
...
)
[partitioned BY (<col_name> <data_type>, ...)] 
stored BY '<StorageHandler>' 
WITH serdeproperties (
 ['<property_name>'='<property_value>',...]
) 
location '<oss_location>' 
USING '<jar_name>';

補全OSS外部表分區數據語法

示例：數據準備

示例：通過內置文本數據解析器創建OSS外部表-非分區表

與示例：數據準備中的Demo1/目錄建立映射關系。創建OSS外部表命令示例如下。

create external table if not exists mc_oss_csv_external1
(
vehicleId int,
recordId int,
patientId int,
calls int,
locationLatitute double,
locationLongtitue double,
recordTime string,
direction string
)
stored by 'com.aliyun.odps.CsvStorageHandler' 
with serdeproperties (
 'odps.properties.rolearn'='acs:ram::xxxxxx:role/aliyunodpsdefaultrole'
) 
location 'oss://oss-cn-hangzhou-internal.aliyuncs.com/oss-mc-test/Demo1/';

您可以執行desc extended mc_oss_csv_external1;命令查看創建好的OSS外部表結構信息。

示例：通過內置文本數據解析器創建OSS外部表-分區表

與示例：數據準備中的Demo2/目錄建立映射關系。創建OSS外部表并引入分區數據命令示例如下。

create external table if not exists mc_oss_csv_external2
(
vehicleId int,
recordId int,
patientId int,
calls int,
locationLatitute double,
locationLongtitue double,
recordTime string
)
partitioned by (
direction string
)
stored by 'com.aliyun.odps.CsvStorageHandler' 
with serdeproperties (
 'odps.properties.rolearn'='acs:ram::xxxxxx:role/aliyunodpsdefaultrole'
) 
location 'oss://oss-cn-hangzhou-internal.aliyuncs.com/oss-mc-test/Demo2/';
--引入分區數據。
msck repair table mc_oss_csv_external2 add partitions;
--等效于如下語句。
alter table mc_oss_csv_external2 add partition (direction = 'N') partition (direction = 'NE') partition (direction = 'S') partition (direction = 'SW') partition (direction = 'W');

您可以執行desc extended mc_oss_csv_external2;命令查看創建好的外部表結構信息。

示例：通過內置文本數據解析器創建OSS外部表-壓縮數據

與示例：數據準備中的Demo3/目錄建立映射關系。創建OSS外部表命令示例如下。

create external table if not exists mc_oss_csv_external3
(
vehicleId int,
recordId int,
patientId int,
calls int,
locationLatitute double,
locationLongtitue double,
recordTime string,
direction string
)
stored by 'com.aliyun.odps.CsvStorageHandler' 
with serdeproperties (
 'odps.properties.rolearn'='acs:ram::xxxxxx:role/aliyunodpsdefaultrole',
 'odps.text.option.gzip.input.enabled'='true',
 'odps.text.option.gzip.output.enabled'='true' 
) 
location 'oss://oss-cn-hangzhou-internal.aliyuncs.com/oss-mc-test/Demo3/';

您可以執行desc extended mc_oss_csv_external3;命令查看創建好的外部表結構信息。

示例：通過內置開源數據解析器創建OSS外部表

• 映射TEXTFILE數據文件

  • 關聯TEXT數據建表示例

    CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_extable_name>
    (
    <col_name> <data_type>,
    ...
    )
    [partitioned BY (<col_name> <data_type>, ...)]
    row format serde 'org.apache.hive.hcatalog.data.JsonSerDe'
    stored AS textfile
    location '<oss_location>';
    SELECT ... FROM <mc_oss_extable_name> ...;

    建表時不支持自定義row format。row format默認值如下。

    FIELDS TERMINATED BY ：'\001'
    ESCAPED BY ：'\'
    COLLECTION ITEMS TERMINATED BY ：'\002'
    MAP KEYS TERMINATED BY ：'\003'
    LINES TERMINATED BY ：'\n'
    NULL DEFINED AS ：'\N'

  • 關聯CSV數據建表示例

    --關閉native的text reader。
    SET odps.ext.hive.lazy.simple.serde.native=false;
    --創建OSS外部表。
    CREATE EXTERNAL TABLE <mc_oss_extable_name>
    (
    <col_name> <data_type>,
    ...
    )
    row format serde 'org.apache.hadoop.hive.serde2.OpenCSVSerde'
    WITH serdeproperties (
    "separatorChar" = ",",
    "quoteChar"= '"',
    "escapeChar"= "\\"
    )
    stored AS textfile
    location '<oss_location>'
    tblproperties (
    "skip.header.line.count"="1",
    "skip.footer.line.count"="1"
    );
    SELECT ... FROM <mc_oss_extable_name> ...;

    說明

    OpenCSVSerde只支持STRING類型。

  • 關聯JSON數據建表示例

    CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_extable_name>
    (
    <col_name> <data_type>,
    ...
    )
    [partitioned BY (<col_name> <data_type>, ...)]
    row format serde 'org.apache.hive.hcatalog.data.JsonSerDe'
    stored AS textfile
    location '<oss_location>';
    SELECT ... FROM <mc_oss_extable_name> ...;

• 映射PARQUET數據文件

  CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_extable_name>
  (
  <col_name> <data_type>,
  ...
  )
  [partitioned BY (<col_name> <data_type>, ...)]
  stored AS parquet  
  location '<oss_location>';
  SELECT ... FROM <mc_oss_extable_name> ...;

• 映射ORC數據文件

  CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_extable_name>
  (
  <col_name> <data_type>,
  ...
  )
  [partitioned BY (<col_name> <data_type>, ...)]
  stored AS orc
  location '<oss_location>';
  SELECT ... FROM <mc_oss_extable_name> ...;

  說明

  具體示例，請參見OSS外部表根據字段名稱Mapping數據。

• 映射RCFILE數據文件

  CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_rcfile_extable>
  (
  <col_name> <data_type>,
  ...
  )
  [partitioned BY (<col_name> <data_type>, ...)]
  row format serde 'org.apache.hadoop.hive.serde2.columnar.LazyBinaryColumnarSerDe'
  stored AS rcfile
  location '<oss_location>';
  SELECT ... FROM <mc_oss_rcfile_extable> ...;

• 映射AVRO數據文件

  CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_extable_name>
  (
  <col_name> <data_type>,
  ...
  )
  [partitioned BY (<col_name> <data_type>, ...)]
  stored AS avro
  location '<oss_location>';
  SELECT ... FROM <mc_oss_extable_name> ...;

• 映射SEQUENCEFILE數據文件

  CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_extable_name>
  (
  <col_name> <data_type>,
  ...
  )
  [partitioned BY (<col_name> <data_type>, ...)]
  stored AS sequencefile
  location '<oss_location>';
  SELECT ... FROM <mc_oss_extable_name> ...;

• 映射Hudi格式數據文件

  說明

  • Hudi外表只支持讀取外表映射的全部文件的數據，不支持自動隱藏系統列，不支持增量讀、快照讀操作，不支持寫操作。建議您使用MaxCompute Delta表或Paimon外部表等功能實現ACID方式的讀寫操作。

  • MaxCompute系統默認集成的Hudi SDK版本為org.apache.hudi:hudi-hadoop-mr-bundle:0.12.2-emr-1.0.6，無法保證Hudi SDK向前或向后兼容，兼容性由開源社區保證。

  • MaxCompute將提供支持用戶上傳jar并指定SDK兼容版本的方式創建外表，用于讀取兼容特定版本的數據。

  CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_extable_name>
  (
  <col_name> <data_type>,
  ...
  )
  [partitioned BY (<col_name> <data_type>, ...)]
  ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
  stored AS 
  INPUTFORMAT 'org.apache.hudi.hadoop.HoodieParquetInputFormat'
  OUTPUTFORMAT 'org.apache.hadoop.hive.ql.io.parquet.MapredParquetOutputFormat'
  location '<oss_location>';
  SELECT ... FROM <mc_oss_extable_name> ...;

• 映射Delta Lake格式數據文件

  說明

  • Delta Lake外表只支持讀取外表映射的全部文件的數據，不支持自動隱藏系統列，不支持增量讀、快照讀操作，不支持寫操作。建議您使用MaxCompute Delta表或Paimon外部表等功能實現ACID方式的讀寫操作。

  • MaxCompute系統默認集成的Delta Lake SDK版本為io.delta:delta-core_2.11:0.2.0.5，無法保證Delta Lake SDK向前或向后兼容，兼容性由開源社區保證。

  • MaxCompute將提供支持用戶上傳jar并指定SDK兼容版本的方式創建外表，用于讀取兼容特定版本的數據。

  CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_extable_name> (
     <col_name> <data_type>,
     ...
  )
  [partitioned BY (<col_name> <data_type>, ...)]
  ROW FORMAT SERDE 'org.apache.hadoop.hive.ql.io.parquet.serde.ParquetHiveSerDe'
  STORED AS
  INPUTFORMAT 'io.delta.hive.DeltaInputFormat'
  OUTPUTFORMAT 'io.delta.hive.DeltaOutputFormat'
  LOCATION '<oss_location>';
  SELECT ... FROM <mc_oss_extable_name> ...;

• 映射Paimon格式數據文件

  CREATE EXTERNAL TABLE [IF NOT EXISTS] <mc_oss_extable_name>
  (
  <col_name> <data_type>,
  ...
  )
  [partitioned BY (<col_name> <data_type>, ...)]
  stored BY 'org.apache.paimon.hive.PaimonStorageHandler'
  WITH serdeproperties (
      'odps.properties.rolearn'='acs:ram::xxxxxxxxxxxxx:role/aliyunodpsdefaultrole'
  )
  location '<oss_location>'
  USING 'paimon_maxcompute_connector.jar';

  paimon_maxcompute_connector.jar需要預先上傳至MaxCompute項目，詳情請參見Paimon外部表。

示例：通過自定義解析器創建OSS外部表

與示例：數據準備中的SampleData/目錄建立映射關系。

操作流程如下：

1. 使用MaxCompute Studio開發TextExtractor.java、TextOutputer.java、SplitReader.java和TextStorageHandler.java四個Java程序。

   更多開發Java程序信息，請參見開發UDF。

2. 通過MaxCompute Studio的一鍵式打包功能，將TextStorageHandler.java打包上傳為MaxCompute資源。

   假設，此處打包的資源名稱為javatest-1.0-SNAPSHOT.jar。更多打包上傳信息，請參見打包、上傳及注冊。

   說明

   如果有多個依賴請分別打包后上傳為MaxCompute資源。

3. 執行如下命令創建OSS外部表。

   CREATE EXTERNAL TABLE [IF NOT EXISTS] ambulance_data_txt_external
   (
   vehicleId int,
   recordId int,
   patientId int,
   calls int,
   locationLatitute double,
   locationLongtitue double,
   recordTime string,
   direction string
   )
   stored BY 'com.aliyun.odps.udf.example.text.TextStorageHandler' 
     WITH serdeproperties (
   'delimiter'='|',  
   'odps.properties.rolearn'='acs:ram::xxxxxxxxxxxxx:role/aliyunodpsdefaultrole'
   )
   location 'oss://oss-cn-hangzhou-internal.aliyuncs.com/oss-mc-test/SampleData/'
   USING 'javatest-1.0-SNAPSHOT.jar'; 

   其中delimiter是OSS數據的分隔符名稱。

   您可以執行desc extended ambulance_data_txt_external;命令查看創建好的外部表結構信息。

   說明

   對于自定義解析器，默認不會對數據分片，防止解析器出現正確性問題。如果您確認可以處理分片，需要使用如下命令允許進行數據分片，即啟動多個mapper。

   set odps.sql.unstructured.data.single.file.split.enabled=true;

示例：通過自定義解析器創建OSS外部表-非文本數據

示例：創建OSS外部表并指定對應OSS文件的第一行為表頭

--創建外部表
CREATE EXTERNAL TABLE mf_oss_wtt
(
  id bigint,
  name string,
  tran_amt double
)
STORED BY 'com.aliyun.odps.CsvStorageHandler'
WITH serdeproperties (
  'odps.text.option.header.lines.count' = '1',
  'odps.sql.text.option.flush.header' = 'true'
)
location 'oss://oss-cn-beijing-internal.aliyuncs.com/mfosscostfee/demo11/';

--插入數據
INSERT overwrite TABLE mf_oss_wtt VALUES (1, 'val1', 1.1),(2, 'value2', 1.3);
--查詢數據
--在建表的時候可以把所有字段建成string，否則表頭讀取時會報錯。
--或者在建表的時候需要加跳過表頭的參數：'odps.text.option.header.lines.count' = '1'
SELECT * FROM mf_oss_wtt;
+------------+------+------------+
| id         | name | tran_amt   |
+------------+------+------------+
| 1          | val1 | 1.1        |
| 2          | value2 | 1.3        |
+------------+------+------------+

示例：創建OSS外部表且外部表列數與OSS數據列數不一致

1. 準備如下數據，保存為CSV格式上傳至OSS的doc-test-01/demo目錄下。

   1,kyle1,this is desc1
   2,kyle2,this is desc2,this is two
   3,kyle3,this is desc3,this is three, I have 4 columns

2. 創建外部表。

   • 指定對于列數不一致行的處理方式為truncate。

     --刪除表
     DROP TABLE test_mismatch;
     --新建外部表
     CREATE EXTERNAL TABLE IF NOT EXISTS test_mismatch
     (
       id string,
       name string,
       dect string,
       col4 string
     )
     stored BY 'com.aliyun.odps.CsvStorageHandler'
     WITH serdeproperties ('odps.sql.text.schema.mismatch.mode' = 'truncate')
     location 'oss://oss-cn-shanghai-internal.aliyuncs.com/doc-test-01/demo/';

   • 指定對于列數不一致行的處理方式為ignore。

     --刪除表
     DROP TABLE test_mismatch01;
     --新建外部表
     CREATE EXTERNAL TABLE IF NOT EXISTS test_mismatch01
     (
       id string,
       name string,
       dect string,
       col4 string
     )
     stored BY 'com.aliyun.odps.CsvStorageHandler'
     WITH serdeproperties ('odps.sql.text.schema.mismatch.mode' = 'ignore'）
     location 'oss://oss-cn-shanghai-internal.aliyuncs.com/doc-test-01/demo/';

3. 查詢表數據。

   • SELECT * FROM test_mismatch;
     
     --返回結果
     +----+------+------+------+
     | id | name | dect | col4 |
     +----+------+------+------+
     | 1  | kyle1 | this is desc1 | NULL |
     | 2  | kyle2 | this is desc2 | this is two |
     | 3  | kyle3 | this is desc3 | this is three |
     +----+------+------+------+

   • SELECT * FROM test_mismatch01;
     
     --返回結果
     +----+------+------+------+
     | id | name | dect | col4 |
     +----+------+------+------+
     | 2  | kyle2 | this is desc2 | this is two |
     +----+------+------+------+

參考：語法參數說明

上述語法中各參數的含義如下。

參考：with serdeproperties屬性列表

參考：tblproperties屬性列表

相關文檔

創建OSS外部表后，您可以通過外部表讀取存儲在OSS目錄中的數據文件。具體操作請參見讀取OSS數據。