1. Introduction
In this tutorial, we go over TASTy (Typed Abstract Syntax Trees), the importance in the compiler toolchain, the compiler directives we use to manipulate TASTy files, and compile-time TASTy reflection. TASTy files (.tasty) are generated and used by the compiler to store program information not available in low-level .class files designed for the JVM.
During the transition from Scala 2 to Scala 3, a new compiler was developed from scratch, called Dotty, utilizing a new TASTy format. Understanding Scala 3 TASTy files will give us a better understanding of compiler internals and the abstract representation of our programs in Scala 3.
2. TASTy Files
When we compile a Scala program, TASTy files are generated as an abstract representation of the source code, including the syntax, typing, and documentation.
Let’s consider a simple program:
final val TASTy: String = "TASTy"
@main def main: Unit = s"Mmm, $TASTy"
If we place the program in the file TASTy.scala and compile it:
$ scalac TASTy.scala
$ ls
main.class 'TASTy$package.tasty'
main.tasty TASTy.scala
'TASTy$package$.class' 'TASTy$package.class'
We see two .tasty files generated.
2.1. Compiling TASTy Files
We can compile .tasty files into .class files without source code using the -from-tasty directive of the compiler:
$ scalac -from-tasty main.tasty
2.2. Decompiling TASTy Files
We can decompile .tasty files back into Scala source code using the compiler:
$ scalac -decompile 'TASTy$package.tasty'
/** Decompiled from ./TASTy$package.tasty */
@scala.annotation.internal.SourceFile("TASTy.scala") object TASTy$package {
final val TASTy: scala.Predef.String = "TASTy"
@scala.main def main: scala.Unit = {
_root_.scala.StringContext.apply("Mmm, ", "").s(TASTy$package.TASTy)
()
}
}
$ scalac -decompile 'main.tasty'
/** Decompiled from ./main.tasty */
@scala.annotation.internal.SourceFile("TASTy.scala") final class main() {
def main(args: scala.Array[java.lang.String]): scala.Unit = try TASTy$package.main catch {
case error: scala.util.CommandLineParser.ParseError =>
scala.util.CommandLineParser.showError(error)
}
}
This is what the code looks like without syntactic sugar.
2.3. Print Tasty Directive
We can also print a TASTy file using a different compiler directive:
$ scalac -print-tasty 'TASTy$package.tasty'
--------------------------------------------------------------------------------
TASTy$package.tasty
--------------------------------------------------------------------------------
Names:
0: ASTs
1: <empty>
2: TASTy$package
3: TASTy$package[ModuleClass]
4: <init>
5: <init>[Signed Signature(List(),TASTy$package$) @<init>]
6: java
7: lang
8: java[Qualified . lang]
9: Object
10: java[Qualified . lang][Qualified . Object]
11: <init>[Signed Signature(List(),java.lang.Object) @<init>]
12: _
13: Unit
14: scala
15: writeReplace
16: AnyRef
17: runtime
18: scala[Qualified . runtime]
19: ModuleSerializationProxy
20: scala[Qualified . runtime][Qualified . ModuleSerializationProxy]
21: Class
22: java[Qualified . lang][Qualified . Class]
23: <init>[Signed Signature(List(java.lang.Class),scala.runtime.ModuleSerializationProxy) @<init>]
24: TASTy
25: String
26: Predef
27: main
28: s
29: java[Qualified . lang][Qualified . String]
30: collection
31: scala[Qualified . collection]
32: immutable
33: scala[Qualified . collection][Qualified . immutable]
34: Seq
35: scala[Qualified . collection][Qualified . immutable][Qualified . Seq]
36: s[Signed Signature(List(scala.collection.immutable.Seq),java.lang.String) @s]
37: apply
38: StringContext
39: scala[Qualified . StringContext]
40: apply[Signed Signature(List(scala.collection.immutable.Seq),scala.StringContext) @apply]
41: _root_
42: StringContext[ModuleClass]
43: Mmm,
44:
45: <repeated>
46: Any
47: scala[Qualified . main]
48: <init>[Signed Signature(List(),scala.main) @<init>]
49: SourceFile
50: annotation
51: scala[Qualified . annotation]
52: internal
53: scala[Qualified . annotation][Qualified . internal]
54: scala[Qualified . annotation][Qualified . internal][Qualified . SourceFile]
55: <init>[Signed Signature(List(java.lang.String),scala.annotation.internal.SourceFile) @<init>]
56: TASTy.scala
57: Positions
58: Comments
The first we see are names, which are indexed and unique identifiers found in our program. Sometimes the name of our value is changed, like when private members need to be accessed from inheriting classes or in nested classes.
Then we see the tree structure of the file:
Trees:
start = Addr(0), base = 456, current = Addr(0), end = Addr(217)
217 bytes of AST, base = Addr(0)
0: PACKAGE(214)
3: TERMREFpkg 1 [<empty>]
5: VALDEF(20) 2 [TASTy$package]
8: IDENTtpt 3 [TASTy$package[ModuleClass]]
10: TYPEREFsymbol 27
12: THIS
13: TYPEREFpkg 1 [<empty>]
15: APPLY(8)
17: SELECTin(6) 5 [<init>[Signed Signature(List(),TASTy$package$) @<init>]]
20: NEW
21: SHAREDterm 8
23: SHAREDtype 10
25: OBJECT
26: SYNTHETIC
27: TYPEDEF(187) 3 [TASTy$package[ModuleClass]]
31: TEMPLATE(161)
34: APPLY(10)
36: SELECTin(8) 11 [<init>[Signed Signature(List(),java.lang.Object) @<init>]]
39: NEW
40: TYPEREF 9 [Object]
42: TERMREFpkg 8 [java[Qualified . lang]]
44: SHAREDtype 40
46: SELFDEF 12 [_]
48: SINGLETONtpt
49: TERMREFsymbol 5
51: SHAREDtype 12
53: DEFDEF(6) 4 [<init>]
56: EMPTYCLAUSE
57: TYPEREF 13 [Unit]
59: TERMREFpkg 14 [scala]
61: DEFDEF(23) 15 [writeReplace]
64: EMPTYCLAUSE
65: TYPEREF 16 [AnyRef]
67: SHAREDtype 59
69: APPLY(13)
71: SELECTin(8) 23 [<init>[Signed Signature(List(java.lang.Class),scala.runtime.ModuleSerializationProxy) @<init>]]
74: NEW
75: TYPEREF 19 [ModuleSerializationProxy]
77: TERMREFpkg 18 [scala[Qualified . runtime]]
79: SHAREDtype 75
81: CLASSconst
82: SHAREDtype 49
84: PRIVATE
85: SYNTHETIC
86: VALDEF(12) 24 [TASTy]
89: IDENTtpt 25 [String]
91: TYPEREF 25 [String]
93: TERMREF 26 [Predef]
95: SHAREDtype 59
97: STRINGconst 24 [TASTy]
99: FINAL
100: DEFDEF(93) 27 [main]
103: IDENTtpt 13 [Unit]
105: TYPEREF 13 [Unit]
107: TERMREFpkg 14 [scala]
109: BLOCK(63)
111: UNITconst
112: APPLY(60)
114: SELECTin(38) 36 [s[Signed Signature(List(scala.collection.immutable.Seq),java.lang.String) @s]]
117: APPLY(31)
119: SELECTin(11) 40 [apply[Signed Signature(List(scala.collection.immutable.Seq),scala.StringContext) @apply]]
122: SELECT 38 [StringContext]
124: SELECT 14 [scala]
126: TERMREFpkg 41 [_root_]
128: TYPEREF 42 [StringContext[ModuleClass]]
130: SHAREDtype 59
132: TYPED(16)
134: REPEATED(6)
136: SHAREDtype 91
138: STRINGconst 43 [Mmm, ]
140: STRINGconst 44 []
142: APPLIEDtype(6)
144: TYPEREF 45 [<repeated>]
146: SHAREDtype 59
148: SHAREDtype 91
150: TYPEREF 38 [StringContext]
152: SHAREDtype 59
154: TYPED(18)
156: REPEATED(8)
158: TYPEREF 46 [Any]
160: SHAREDtype 59
162: TERMREFsymbol 86
164: SHAREDtype 49
166: APPLIEDtype(6)
168: SHAREDtype 144
171: SHAREDtype 158
174: ANNOTATION(19)
176: TYPEREF 27 [main]
178: SHAREDtype 59
180: APPLY(13)
182: SELECTin(11) 48 [<init>[Signed Signature(List(),scala.main) @<init>]]
185: NEW
186: IDENTtpt 27 [main]
188: TYPEREF 27 [main]
190: SHAREDtype 107
192: SHAREDtype 176
195: OBJECT
196: SYNTHETIC
197: ANNOTATION(18)
199: TYPEREF 49 [SourceFile]
201: TERMREFpkg 53 [scala[Qualified . annotation][Qualified . internal]]
203: APPLY(12)
205: SELECTin(8) 55 [<init>[Signed Signature(List(java.lang.String),scala.annotation.internal.SourceFile) @<init>]]
208: NEW
209: SHAREDtype 199
212: SHAREDtype 199
215: STRINGconst 56 [TASTy.scala]
And finally some other metadata:
101 position bytes:
lines: 4
line sizes: 33, 0, 37, 0
positions:
0: 0 .. 72
5: 0 .. 0
8: 0 .. 0
27: 0 .. 72
31: 0 .. 72
40: 0 .. 0
49: 0 .. 0
53: 0 .. 0
57: 0 .. 0
61: 0 .. 0
65: 0 .. 0
75: 0 .. 0
81: 0 .. 0
86: 0 .. 33
89: 17 .. 23
97: 26 .. 33
100: 35 .. 72
103: 51 .. 55
111: 72 .. 72
114: 58 .. 72
126: 60 .. 60
136: 71 .. 71
138: 60 .. 65
140: 71 .. 71
142: 71 .. 71
158: 71 .. 71
162: 66 .. 71
166: 71 .. 71
185: 35 .. 40
186: 36 .. 40
203: 0 .. 72
209: 0 .. 0
215: 0 .. 0
source paths:
0: TASTy.scala
0 comment bytes:
4. TASTy Inspection
If accessing the TASTy structure through program reflection is more convenient than in a terminal, the Scala compiler team has a library that allows direct inspection of the TASTy structure called scala3-tasty-inspector. We can add the library to an SBT project with this artifact:
libraryDependencies += "org.scala-lang" %% "scala3-tasty-inspector" % scala3Version
Or we can use the official pre-made template for TASTy inspection:
$ sbt new scala/scala3-tasty-inspector.g8
To test it, we can define a package:
package lib
object Greetings {
def hello(name: String): String = "Hello " + name
}
Then we define another package we use to inspect its structure:
object Inspector:
def showCodeOf(tastyFile: String): String =
var tastyStr: String = null
val inspector = new Inspector:
def inspect(using Quotes)(tastys: List[Tasty[quotes.type]]): Unit =
import quotes.reflect.*
for tasty <- tastys do
val projectDir = new java.io.File( "." ).getCanonicalPath() + java.io.File.separator // To cleanup the paths in @SourceFile
tastyStr = tasty.ast.show.replace(projectDir, "")
TastyInspector.inspectTastyFiles(List(tastyFile))(inspector)
tastyStr
@main def run =
val tastyFile = "lib/target/scala-3.4.2/classes/lib/Greetings.tasty"
val tastyContents = Inspector.showCodeOf(tastyFile)
println(tastyContents)
When we run the inspector package, we print the decompiled contents of the lib package:
[info] running inspector.run
package lib {
@scala.annotation.internal.SourceFile("lib/src/main/scala/lib/Greetings.scala") object Greetings {
def hello(name: scala.Predef.String): scala.Predef.String = "Hello ".+(name)
}
}
The Inspector gives us access to the TASTy file as an AST. Tasty is a trait with two abstract methods:
trait Tasty[Q <: Quotes & Singleton]:
def ast: Tree
def path: String
Tree models the AST of any Scala 3 program:
We can inspect and manipulate the AST accordingly.
5. Macros
Scala metaprogramming has been revamped with new macro capabilities built on top of TASTy inspection. Instead of inspecting .tasty files, we use quoted expressions and types from the quoted package:
import scala.quoted.*
object PowerMacro:
// The macro that unrolls the computation of powers and then generates the expression
inline def showAsPowerTerm(inline x: Double, n: Int): String = ${ showAsTermImpl('x, 'n) }
// The actual implementation of the macro
private def showAsTermImpl(x: Expr[Double], n: Expr[Int])(using Quotes): Expr[String] =
import quotes.reflect.*
n.value match
case Some(num) =>
val powerExpr = unrolledPowerCode(x, num)
Expr(powerExpr.asTerm.toString) // Ensures that the asTerm method call is evaluated at compile-time
case None =>
'{ "Error: 'n' must be a known constant at compile time." }
// Helper method to unroll the power computation
def unrolledPowerCode(x: Expr[Double], n: Int)(using Quotes): Expr[Double] =
if n == 0 then '{ 1.0 }
else if n == 1 then x
else '{ $x * ${ unrolledPowerCode(x, n-1) } }
We can then use showAsPowerTerm to generate the unrolledPowerCode AST at compile-time:
@main def runMacro(): Unit =
val expr: String = PowerMacro.showAsPowerTerm(2.0, 3)
println(expr)
The output is the term structure generated by our macro at compile-time:
Inlined(
Ident(PowerMacro$),
List(),
Apply(
Select(
Inlined(
EmptyTree,
List(),
Inlined(
EmptyTree,
List(),
Literal(Constant(2.0))
)
),
'*'
),
List(
Inlined(
EmptyTree,
List(),
Inlined(
Ident(PowerMacro$),
List(),
Apply(
Select(
Inlined(
EmptyTree,
List(),
Inlined(
EmptyTree,
List(),
Literal(Constant(2.0))
)
),
'*'
),
List(
Inlined(
EmptyTree,
List(),
Inlined(
EmptyTree,
List(),
Literal(Constant(2.0))
)
)
)
)
)
)
)
)
)
TASTy reflection on quotations allows us to inspect and manipulate the TASTy structure of our program at compile-time.
6. Conclusion
In this article, we reviewed the importance of TASTy files in the Dotty (Scala 3) compiler toolchain, the basic compiler commands we use to create and understand .tasty files (including -print-tasty, -from-tasty, and -decompile), and the naming and value referencing model.
As usual, the code is available over on GitHub.