Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ Reading a log from a U-SQL Script



To use the Extractors, you U-SQL script will need to have these statements and
To use the Extractors, you U-SQL script will need to have these statements at
the beginning:


Expand Down
2 changes: 1 addition & 1 deletion docs/Hands_on_Labs/ADL_Store_CLI_HOL.md
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,7 @@ In this exercise you will list files within your Data Lake Store account, you'll
2. Create a new file within the root directory of your Data Lake Store account.
* Use the command ``azure datalake store filesystem create``.

3. Add content to to the newly created file.
3. Add content to the newly created file.
* Use the command ``azure datalake store filesystem addcontent``.

# Exercise 3: Reading data from Data Lake Store:
Expand Down
4 changes: 2 additions & 2 deletions docs/Hands_on_Labs/Start.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ portal
- Pick a Location (Region) to use. All the Data Lake services should be in the
same location. The Azure Data Factory account can be in any region

- You can choose any Azure Resource Group that you want. It won’t affect the
- You can choose any Azure Resource Group that you want. It won’t affect
how the labs are done. We recommend using one like ....

"username_adl_hol"
Expand Down Expand Up @@ -84,7 +84,7 @@ Go to http://portal.azure.com

- Click **Create**

### Step 3: Create a Azure Data Factory Account
### Step 3: Create an Azure Data Factory Account

- Click **New \> Data + Analytics \> Data Factory**

Expand Down
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@

Azure Data Lake > U-SQL Release Notes 2016/02/16
Azure Data Lake > U-SQL Release Notes 2016/02/15
--------------------------

# BREAKING CHANGES
Expand All @@ -18,7 +18,7 @@ Previously, User-defined types (UDTs) in the EXTRACT/PRODUCE clauses are bound a

1. Starting with this release, UDTs in EXTRACT/PRODUCE clauses are bound only against the referenced assemblies in the current script/code object.

2. For example, if you used a UDT in a VIEW, FUNCTION or PROCEDURE body, they were visible inside. Now they will not visible. To make them visible, please move the REFERENCE assembly statement into the body of a FUNCTION or PROCEDURE. In case of a VIEW, please rewrite the view as a table-valued function.
2. For example, if you used a UDT in a VIEW, FUNCTION or PROCEDURE body, they were visible inside. Now they will not be visible. To make them visible, please move the REFERENCE assembly statement into the body of a FUNCTION or PROCEDURE. In case of a VIEW, please rewrite the view as a table-valued function.

3. As a consequence of changes 1 and 2, VIEW definitions cannot contain any user-defined functions (UDF), types (UDT), aggregators (UDAGG), nor operators (UDO). This is because the body of a VIEW can only contain a single query statement and no REFERENCE ASSEMBLY statements. Thus a VIEW's body is always resolved against system assemblies only. Please rewrite any view that needs to refer to a user-defined object (UDx) into a table-valued function.

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ with this release and the requirement to add a predicate will be dropped:

2. _A future release_ (probably before the GA release): U-SQL will remove `{col:*}` and will only support `{col}`.

So please start to replace your {col:*} with {col}. Note that it is still good practise to add a predicate on such columns to reduce the number of processed files.
So please start to replace your {col:*} with {col}. Note that it is still good practice to add a predicate on such columns to reduce the number of processed files.

We will announce the future phase out on our blog/release notes.

Expand All @@ -27,21 +27,21 @@ As an aside: The `{*}` pattern is still supported for unnamed wildcards.
#### Space encoding in file paths are fixed

In previous releases, specifying paths with special characters such as spaces were causing the following problems:
- Files paths with spaces were not found in file set specifications.
- File paths with spaces were not found in file set specifications.
- Some file locations would work with URI escaped paths but others would not.

These issues have been fixed:
- File Set wildcards will match file paths with special characters in file paths such as spaces and will expose
them in the virtual column unescaped, e.g. a virtual column pattern matching the file part "filename with spaces"
will now will show "filename with spaces".
will now show "filename with spaces".
- In order to specify spaces in file paths, the provided file path has to be specified with the standard
URL encoding for spaces: `%20`. For example, if you want to `EXTRACT` from a file "filename with spaces.csv" you write
`FROM "filename%20with%20spaces.csv"`.

#### Data corruption due to non-determinism during writing into a U-SQL table is fixed

Under some conditions such as a plan that contains vertices that write to a table and any other location (including another vertex, table or file),
and one of these vertices has to be rerun, there is a chance that the table meta data was not correctly updated.
and one of these vertices has to be rerun, there is a chance that the table metadata was not correctly updated.
This would manifest itself in corrupted, non-readable tables. With this release, this condition has been addressed.

## New U-SQL capabilities
Expand All @@ -61,7 +61,7 @@ Alter_Table_AddDrop_Column_Statement :=
( 'ADD' 'COLUMN' Column_Definition_List
| 'DROP' 'COLUMN' Identifier_List ).
````
Adding or dropping a column from a table is a meta data operation only and its performance will not be impacted by the size of the table. If the added column is of a nullable type, existing rows will contain null in the added column, if the added column is of a not-nullable type, then the column will contain the type's default value (e.g., `0` for type `int`).
Adding or dropping a column from a table is a metadata operation only and its performance will not be impacted by the size of the table. If the added column is of a nullable type, existing rows will contain null in the added column, if the added column is of a not-nullable type, then the column will contain the type's default value (e.g., `0` for type `int`).

Examples:
````
Expand Down Expand Up @@ -102,7 +102,7 @@ Note that these declarations are not constant-foldable.
#### U-SQL now supports a `USING` directive that allows shortening U-SQL custom code object references.

The new `USING` directive is similar to the C# `using` directive in that it allows to shorten function,
type, extention methods, and UDO path names.
type, extension methods, and UDO path names.

Syntax:

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

Previously, we had account level access control for ADLS. In this release, we give users granular control over the access list for individual folders and files. Simply browse to the folder or file you want to set the access control on and click on the "Access" command.

For greater compatilibity with HDFS, we've implemented the full POSIX ACL system. More details can be found in the blog post [here](https://blogs.msdn.microsoft.com/azuredatalake/2016/07/31/introducing-file-and-folder-acls-for-azure-data-lake-store/)
For greater compatibility with HDFS, we've implemented the full POSIX ACL system. More details can be found in the blog post [here](https://blogs.msdn.microsoft.com/azuredatalake/2016/07/31/introducing-file-and-folder-acls-for-azure-data-lake-store/)

## Securing Catalogs and Databases in Azure Data Lake Analytics

Expand All @@ -23,5 +23,5 @@ To manage permissions on a Catalog/Database:

## Simplified adding a new user to Azure Data Lake Analytics

With the increased in control over Catalog, Database, Folder, and File access, additional steps are required to give access to the dependencies ADLA needs to submit a job. We added an "Add New User" wizard in Azure Data Lake Analytics to automate the process needed to add a new user to submit a job.
With the increase in control over Catalog, Database, Folder, and File access, additional steps are required to give access to the dependencies ADLA needs to submit a job. We added an "Add New User" wizard in Azure Data Lake Analytics to automate the process needed to add a new user to submit a job.
To run the wizard, simply click on the "Add User Wizard" button in the ADLA overview.
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,13 @@

## Latest Component Changes

Azure Data Lake Analytics (ADLA) is updated on an aperiodic basis. We continue to provide the support for this service with certain component update when needed, e.g.: certain shared component update along with other analytic service and so on.
Azure Data Lake Analytics (ADLA) is updated on an aperiodic basis. We continue to provide the support for this service with certain component updates when needed, e.g.: certain shared component update along with other analytic service and so on.

Check the below content for the component changes in recent update.

#### 1. .NET Framework upgraded from 4.5.2 to 4.7.2

The Azure Data Lake Analytics default runtime .Net Framework has been upgraded from .NET Framework v4.5.2 to .NET Framework v4.7.2. This upgrade from .NET Framework 4.5.2 to version 4.7.2 means that the .NET Framework deployed in a U-SQL runtime (the default runtime) will now always be 4.7.2. (Note: the actual upgrade happens several months ago. This is to record this change in ADL release note for history check.)
The Azure Data Lake Analytics default runtime .Net Framework has been upgraded from .NET Framework v4.5.2 to .NET Framework v4.7.2. This upgrade from .NET Framework 4.5.2 to version 4.7.2 means that the .NET Framework deployed in a U-SQL runtime (the default runtime) will now always be 4.7.2. (Note: the actual upgrade happened several months ago. This is to record this change in ADL release note for history check.)

For more detailed information regarding this upgrade, refer to [Azure Data Lake Analytics is upgrading to the .NET Framework v4.7.2](https://docs.microsoft.com/en-gb/azure/data-lake-analytics/dotnet-upgrade-troubleshoot)

Expand Down